AIR  FORCE  PROJECT  RISK  MANAGEMENT  -  THE  IMPACT  OF 
INCONSISTENT  PROCESSES 

THESIS 


Eric  J.  Perez,  Captain,  USAF 

AFIT-ENV-MS- 1 6-S-047 

DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright- Patterson  Air  Force  Base,  Ohio 


DISTRIBUTION  STATEMENT  A. 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  United 
States  Government.  This  material  is  declared  a  work  of  the  U.S.  Government  and  is  not 
subject  to  copyright  protection  in  the  United  States. 


AFIT-ENV-MS-16-S-047 


AIR  FORCE  PROJECT  RISK  MANAGEMENT  -  THE  IMPACT  OF 
INCONSISTENT  PROCESSES 

THESIS 

Presented  to  the 

Department  of  Systems  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Systems  Engineering 


Eric  J.  Perez,  BSSE 
Captain,  USAF 


September  2016 

DISTRIBUTION  STATEMENT  A. 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 


AFIT-ENV-MS-16-S-047 


AIR  FORCE  PROJECT  RISK  MANAGEMENT  -  THE  IMPACT  OF 
INCONSISTENT  PROCESSES 


Eric  J.  Perez,  BS 
Captain,  USAF 


Committee  Membership: 


Lt  Col  K.F.  Oyama 
Chair 


Dr.  R.  David  Fass 
Member 


Dr.  John  Elshaw 
Member 


AFIT-ENV-MS-16-S-047 


Abstract 


On  the  surface,  the  majority  of  Air  Force  units  appear  to  be  executing  some  form 
of  project  risk  management  (PRM).  Deeper  investigation  exposes  many  of  the  PRM 
practices  currently  in  place  to  often  be  ad  hoc,  and  as  such,  have  the  potential  to  poorly 
inform  decisions  when  risk  data  is  escalated  to  the  enterprise  levels  of  the  Air  Force. 
Resultantly,  there  is  inconsistency  between  the  PRM  processes  practiced  by  Air  Force 
units.  This  inconsistency  is  an  indication  of  risk  mishandling  at  the  project  level  and 
erroneous  risk  data  provided  at  the  enterprise  levels  of  the  Air  Force.  It  may  be  the  case 
that  PRM  process  inconsistency  could  be  attributed  to  ambiguous  direction,  cognitive 
biases,  flawed  practices,  and  inadequate  tools.  To  investigate  whether  this  is  the  case,  a 
pool  of  twenty  subjects  who  represented  a  wide  slice  of  Air  Force  acquisitions  personnel 
were  interviewed  to  provide  insight  to  the  underlying  phenomenon  causing  inconsistent 
and  deficient  Air  Force  PRM.  Specifically,  the  data  collected  was  sorted  and  analyzed 
for  trends  and  patterns  concerning  current  Air  Force  PRM  practices.  The  information 
from  the  research  study  can  be  used  as  a  launch  point  to  focus  corrective  efforts  and 
changes  on  the  sources  of  the  problems  identified. 
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AIR  FORCE  PROJECT  RISK  MANAGEMENT  INCONSISTENCY  AND 


POTENTIAL  SOLUTIONS 
I.  Introduction 


Background 

The  study  of  project  risk  management  (PRM)  is  a  mainstay  within  the  mandatory 
educational  framework  studied  by  Air  Force  acquisitions  officers.  As  an  Air  Force 
acquisitions  officer,  the  author  has  received  Air  Force  PRM  training  and  has  been  directly 
involved  with  practicing  PRM  within  multiple  Air  Force  programs.  On  a  basic  level,  the 
practice  of  PRM  is  akin  to  a  person’s  innate  instinct  to  avoid  pain  and  seek  pleasure. 
Comparatively,  with  regard  to  decisions  within  project  development,  people  instinctively 
seek  to  mitigate  the  impact  of  risks  and  capitalize  on  opportunities  to  obtain 
organizational  success.  However,  upon  further  study,  it  became  evident  to  the  author  that 
complex  Air  Force  acquisitions  programs  necessitate  a  proactive  and  structured  approach 
to  risk  management.  If  implemented  properly,  PRM  becomes  the  backbone  for  decision¬ 
making  and  uncertainty  management. 

Per  firsthand  observations  made  by  the  author  in  the  Air  Force  acquisitions 
environment,  the  subject  of  this  thesis  is  concerned  with  ineffective  Air  Force  PRM. 
Specifically,  if  PRM  practices  at  the  project  level  are  inconsistent,  then  subsequent  macro 
or  enterprise  level  decisions  will  be  misinformed.  It  is  likely  that  PRM  processes  are 
inconsistently  practiced  by  units  in  the  Air  Force  because  of  ambiguous  direction, 
cognitive  biases,  flawed  practices,  and  inadequate  tools;  this  would  be  evidence  of  project 
risk  mishandling  and  leads  to  misinformed  decisions  at  the  Air  Force  enterprise  levels. 
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Problem  Statement 


As  with  any  process,  there  is  a  natural  evolution  of  refinement  between  initial 
implementation  and  a  state  of  optimum  performance.  PRM  in  the  Air  Force  is  not  an 
exception,  and  continuous  assessment  must  be  made  to  either  maintain  or  achieve  an 
optimum  state  of  performance.  Analysts  at  the  RAND  Arroyo  Center  for  research  and 
development  established  a  simple  approach  to  controlling  the  maturation  of  a  process, 
referred  to  as  velocity  management,  which  has  been  adopted  and  proven  successful  for 
the  U.S.  Government  (RAND,  2001).  Simply  put,  velocity  management  is  a  method  that 
iteratively  defines  a  process  and  then  uses  measurements  of  performance  to  drive  process 
improvement.  PRM  in  the  Air  Force  must  be  similarly  scrutinized.  A  “measurement”  of 
current  Air  Force  PRM  performance  may  substantiate  the  author’s  observations  of  poor 
Air  Force  PRM  performance  and  assumptions  that  inconsistencies  are  to  blame. 
Regardless  of  whether  the  author’s  assumptions  are  validated  or  not,  a  study  to  evaluate 
Air  Force  PRM  and  the  resulting  data  will  be  essential  to  improving  a  process  that  is 
critical  to  the  decisions  made  at  the  enterprise  level.  To  this  end,  the  study  will 
investigate  the  assumption  that  the  root  cause  for  the  observed  lack  of  PRM  effectiveness 
is  the  combinatory  effect  of  several  factors.  These  factors  are  articulated  below. 

Ambiguous  Direction.  Although  there  seems  to  be  a  general  basic  understanding 
of  an  Air  Force  PRM  concept,  divergence  is  evident  in  multiple  crucial  areas.  Because 
the  Air  Force  has  a  uniquely  high  rate  of  military  leadership  turnover  and  often  integrates 
many  different  personnel  types  together  to  form  acquisition  teams,  there  exists  a  diverse 
set  of  professional  backgrounds.  Diversity  has  the  potential  to  bring  innovation  and 
knowledge  to  project  development,  but  it  can  also  fragment  concepts  like  PRM  where 
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congruency  is  needed.  For  instance,  an  integrated  product  team  (IPT)  can  consist  of 
active-duty  military,  government  civilian,  and  contractor  representatives  who  may  all 
have  a  different  source  of  training  and  directives  for  the  execution  of  PRM.  Officially 
published  guidance  and  documentation  can  be  used  to  normalize  a  single  framework  for 
conducting  PRM  in  Air  Force  acquisitions. 

Although  the  Air  Force  does  have  publications  that  direct  and  guide  PRM,  their 
combined  ambiguity  makes  them  a  weak  point  of  leverage.  As  will  be  argued  by  the 
author,  debates  regarding  how  Air  Force  PRM  publications  should  be  interpreted  are  not 
uncommon.  Additional  sources  of  instruction  extraneous  to  the  Air  Force,  often 
sponsored  by  contractors,  add  to  the  ambiguity  and  can  completely  sideline  Government 
directives.  A  lack  of  clear  Air  Force  PRM  guidance  and  instruction  leads  to  a  varying 
array  of  PRM  methods  and  practices  across  the  Air  Force;  those  with  the  strongest  voice 
for  a  particular  PRM  method  usually  prevail.  Stakeholders  are  averse  to  inconsistent  and 
debatable  processes,  and  the  concept  of  PRM  is  blamed  instead  of  the  cause.  As  such, 
stakeholder  confidence  and  buy-in  must  be  achieved  before  a  process  can  flourish. 

Biases.  Because  PRM  is  not  a  definitive  process,  but  instead  is  one  that  is  based 
on  subjective  estimation,  there  exists  susceptibility  to  various  inconspicuous  influences. 
For  clarity  and  insight  into  the  underlying  phenomenon,  it  is  effective  to  categorize 
influences  into  witting  and  unwitting  biases.  A  witting  bias  in  terms  of  its  effect  on  PRM 
includes  risk  assessment  data  that  is  skewed  in  order  to  fulfill  an  agenda  other  than  the 
pursuit  of  accuracy.  For  instance,  reporting  high  risks  will  most  often  equate  to  time- 
consuming  oversight  and  micromanagement  that  doubly  becomes  a  risk  in  and  of  itself. 

It  is  the  author’s  experience  that  a  program  manager  who  is  already  under  pressure  to 
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meet  cost,  schedule,  and  performance  restraints  will  be  inclined  to  project  the  least  risk  as 
possible.  Another  example  comes  as  the  result  of  a  lack  of  buy-in  to  the  process 
altogether.  Often  referred  to  as  pencil  whipping,  risk  managers  may  be  inclined  to  make 
risk  estimations  with  little  to  no  effort  given  towards  achieving  accuracy.  With  no 
concern  for  accuracy,  there  are  probably  no  reservations  in  skewing  the  data  to  report 
favorably.  Though  the  examples  provided  seem  alarming  and  deceptive,  the  author  has 
witnessed  unwitting  bias  to  be  the  most  prominent  and  detrimental  to  PRM  accuracy. 

Groups,  teams,  even  whole  organizations  are  susceptible  to  falling  under  the  spell 
of  group  behavioral  patterns  without  being  aware.  When  working  together,  a  group  of 
individuals  can  increasingly  diverge  from  rationality  until  an  outsider  or  hindsight  bears 
truth  to  their  reality.  A  common  example  of  this  trend  is  called  groupthink,  which  is 
defined  as  the  tendency  for  group  members  toward  converging  opinions  about  the 
adoption  of  a  certain  course  of  action  in  a  given  decision  situation  (Janis,  1972).  The 
author  has  experienced  firsthand  how  a  group  can  collectively  allow  agreeability  to 
overpower  rationality  for  several  reasons,  such  as  the  persuasion  of  strong  personalities, 
tendency  to  have  an  overly  optimistic  outlook,  and  an  apathetic  desperation  for  resolve. 
The  logical  outcome  of  groupthink  is  that  undetected  influences  caused  by  a  behavioral 
phenomenon  tend  to  proliferate  until  little  to  no  one  is  left  to  see  the  error. 

Human  behavior  will  persist  to  affect  the  good  judgment  of  any  Air  Force  team 
working  towards  a  common  goal.  The  practice  of  PRM  is  especially  susceptible,  because 
predicting  the  future  is  highly  subjective,  erroneous  patterns  of  reason  are  not  definitively 
evident. 
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Flawed  Practices.  Air  Force  PRM,  just  like  any  other  process,  always  has 
potential  to  improve.  The  concern  should  not  be  whether  or  not  the  process  is  currently 
perfect.  Instead,  the  concern  should  be  aligned  with  the  pursuit  of  perfection  through 
strategic  change.  On  one  hand,  it  is  the  author’s  opinion  that  the  Air  Force  practices  that 
are  consistent  have  grown  stale  and  unimproved  despite  identified  deficiencies.  On  the 
other,  the  practices  that  may  be  effective  are  not  consistently  used  and  therefore  do  not 
contribute  to  an  enterprise  level  assessment.  The  author  has  gained  insight  into  how 
competing  companies  proposed  to  conduct  PRM  in  collaboration  with  the  Air  Force. 

One  of  three  approaches  to  PRM  were  characteristically  submitted  for  consideration  by 
various  companies  include  the  following:  a)  PRM  practices  proposed  were  different  than 
Air  Force  practices  and  deemed  by  the  source  selection  team  as  better;  b)  PRM  practices 
proposed  were  the  same  as  Air  Force  practices,  and;  c)  PRM  practices  were  different  than 
Air  Force  practices  and  deemed  by  the  source  selection  team  as  better  and  a  method  to 
mesh  with  standardized  Air  Force  PRM  was  provided.  In  essence,  the  Air  Force  is 
falling  behind  industry  best  practices  despite  having  previously  set  the  benchmark  for 
PRM.  This  is  especially  ironic  when  one  considers  that  in  earlier  years,  Air  Force 
missions  to  spearhead  complex  programs  like  space  access  and  missile  defense 
necessitated  effective  systems  engineering  which  devolved  a  keen  focus  on  PRM. 

There  are,  however,  organizations  within  the  Air  Force  that  are  going  beyond  the 
current  stagnant  standards  and  pursuing  homegrown  best  practices.  Most  commonly,  the 
author  has  observed  nonstandard  PRM  practices  to  be  the  result  of  and  influenced  by  the 
multitude  of  Air  Force  industry  partners  involved  with  any  given  program.  The  pursuit  to 
improve  and  leam  from  others  is  undoubtedly  a  step  in  the  right  direction.  However,  a 


5 


multitude  of  different  influences  results  in  a  multitude  of  different  processes  that 
ultimately  forms  fragmented  data  when  aggregated  for  an  enterprise  level  assessment. 

Inadequate  Tools.  In  the  pursuit  of  global  superiority,  the  U.S.  Air  Force 
involves  itself  with  new  and  complex  acquisitions  and  development.  The  consequent  risk 
and  management  of  that  risk  is  equally  as  complicated  and  therefore  necessitates  tools  to 
augment  the  process.  Having  worked  directly  in  the  field  of  study,  the  author  is  only 
aware  of  one  tool  used  by  the  Air  Force,  Active  Risk  Manager  (ARM),  and  is  described 
by  the  Space  and  Missile  Systems  Center  Risk  Management  Process  Guide  as,  “The  Air 
Forces  current  standard  tool  to  manage  program  risks”  (Conrow  2014,  p.  25).  The 
inadequacy  is  not  due  to  there  being  only  one  tool  observed  since  ARM  strives  to  be  a 
suite  of  tools.  Rather,  it  is  the  suitability  of  ARM  that  seems  to  be  lacking. 

Lack  of  suitability  observed  by  the  author,  is  evidenced  directly  by  its  limited  to 
virtually  non-existent  use  at  the  CGO  level  when  working  on  multiple  complex,  high- 
risk,  critical  Air  Force  systems.  More  specifically,  there  was  non-availability  of  ARM  to 
those  involved  with  PRM  at  the  CGO  level  and  a  lack  of  features  designed  to  manage 
risks  as  opposed  to  just  reporting.  An  analysis  of  risk  management  tools  described  ARM 
as  an  expensive,  standalone  system  with  limited  and  awkward  interfaces  (Langbein, 
2005).  It  is  possible  that  the  high  cost  of  licensing  contributes  to  the  non-availability  of 
ARM  at  the  CGO  level.  Consequently,  the  needs  of  the  limited  group  using  ARM  at  the 
executive  level  might  be  driving  utility  design  to  favor  the  ability  to  report  risks  rather 
than  manage  them.  Furthermore,  the  complexity  of  the  tool  itself  is  overbearing  when 
considering  the  workload  of  a  systems  engineer  or  program  manager  at  the  CGO  level. 
Though  very  capable,  ARM  is  better  suited  for  a  dedicated  PRM  SME,  which  is  a 
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position  rarely  available.  Negatives  aside,  ARM  is  an  excellent  starting  point  to  develop 
a  less  complex  and  more  affordable  add-in  that  lends  itself  to  PRM  at  the  CGO  level. 

In  order  to  increase  PRM  effectiveness  in  the  Air  Force,  deficiencies  with  the 
current  processes  must  be  first  be  identified  and  then  studied.  The  assumptions  made 
thus  far  about  the  contributors  to  the  observed  PRM  deficiencies  will  be  investigated  in 
terms  of  a  set  of  questions.  In  this  effort,  the  following  five  questions  will  serve  as  the 
framework  for  this  thesis: 

•  Is  project  risk  management  (PRM)  inconsistently  practiced  in  the  U.S.  Air  Force? 

•  What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

•  How  do  inconsistencies  in  Air  Force  PRM  processes  lead  to  inaccurate  risk 
assessments? 

•  How  do  inconsistent  PRM  assessments  impact  executive  RM? 

•  What  contributes  to  Air  Force  PRM  inconsistencies? 

•  What  steps  can  be  taken  to  address  contributors  to  inconsistent  PRM  across  the 
Air  Force? 

Research  Methodology 

There  are  multiple  variables  that  shape  the  PRM  practices  used  in  the  Air  Force; 
the  method  used  to  identify  what  contributes  to  the  PRM  inconsistencies  observed  must 
account  for  this  complex  environment.  For  example,  the  author  suspects  that  the  most 
insight  will  come  from  investigating  PRM  practices,  DoD  directives,  organizational 
philosophy,  and  mission  characteristics.  However,  limiting  the  scope  of  the  investigation 
may  miss  key  data.  The  literature  and  author’s  experiences  on  PRM  inconsistencies  will 
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drive  initial  assumptions  and  a  starting  point  for  an  explanation,  but  the  research  method 
used  by  this  study  must  impartially  seek  a  full  and  complete  answer  from  the  field  of 
study.  To  that  end,  this  project  will  use  a  qualitative  grounded  theory  methodology  to 
collect  and  analyze  interview  data  from  a  number  of  research  subjects. 

The  initial  information  found  in  the  literature  review  will  be  combined  with  the 
findings  from  the  research  data  that  will  be  analyzed  for  contributors  to  PRM 
inconsistency  and  the  extent  of  their  impact.  To  do  so,  the  grounded  theory  research 
method  used  in  this  project  must  take  data  from  multiple  sources,  be  amiable  to  new  data, 
and  be  able  to  assess  validity.  All  considering,  a  qualitative  data  gathering  technique  by 
way  of  interviewing  experts  involved  with  Air  Force  acquisitions  will  be  best  suited  to 
discover  the  underlying  phenomenon  that  leads  to  inconsistent  PRM. 

The  interviews  conducted  must  elicit  the  data  needed  to  identify  all  and  any 
contributors  to  Air  force  PRM  inconsistency  through  strategic  questioning  techniques  and 
interviewing  style.  The  interview  question  set,  also  termed  a  schedule,  will  be  semi- 
structured.  In  other  words,  the  schedule  will  consist  of  questions  inspired  by  the  author’s 
initial  observations  and  findings  in  the  relevant  literature  while  also  assuming  an  open- 
ended  format  to  allow  respondents  to  lead  into  discussions  beyond  the  initial 
assumptions.  Ideally,  interviews  will  be  conducted  in  person  but  when  not  possible, 
communication  via  telephone  may  be  necessary  to  reach  key  subject  matter  experts 
(SME).  Building  rapport  is  one  approach  that  will  be  used  to  elicit  valid  information 
from  subjects  and  is  assumed  by  the  author  to  be  easier  achieved  in  person  versus 
telephone  communication.  However,  the  subjects  slated  for  interviewing  will  be  able  to 
relate  to  the  author  in  terms  of  Air  Force  camaraderie,  the  acquisitions  profession,  and  the 
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specific  rigors  of  PRM  deficiencies.  Commonalities  in  this  regard  are  advantageous  in 
building  rapport  and  will  work  to  overcome  any  lack  of  in-person  availability.  When  a 
sufficient  amount  of  raw  data  is  collected,  it  will  be  processed  and  analyzed  for  validity 
and  consistency. 

Processing  raw  data  from  the  interviews  will  require  an  analytical  approach  to 
uncover  themes  that  explain  PRM  inconsistency.  Namely,  the  approach  that  will  be  used 
is  termed  grounded  theory,  in  which  case  the  initial  explanation  derived  from  the  author’s 
experiences  and  review  of  the  literature  will  be  compared  to  the  data  collected  from  the 
field  of  study  for  a  more  complete  answer.  This  approach  is  interested  in  the  aspect  of 
the  problem  as  it  relates  to  project  risk  management  teams  and  the  processes  and  tools 
they  are  bound  to.  The  complexity  experienced  by  the  author  in  Air  Force  acquisitions  is 
predicted  to  translate  through  to  the  responses  received  in  the  interviews.  Once  themes 
are  developed  from  the  data,  the  themes  may  contradict  or  further  bolster  the  author’s 
initial  explanation.  Regardless,  the  risk  of  bias  will  be  acknowledged  and  emphasis  will 
be  placed  on  validating  the  data  for  the  most  accurate  and  complete  depiction  of  the 
underlying  phenomenon  as  possible. 

Assumptions/Limitations 

The  subjective  nature  of  PRM  and  qualitative  research  leaves  potential  for  bias 
and  differing  interpretations  by  the  subjects  interviewed  and  by  the  researcher.  By 
design,  semi-structured  interviews  and  open-ended  questioning  gives  free  reign  for 
subjects  to  express  their  views.  While  doing  so  gives  an  opportunity  for  new  ideas  to  be 
generated,  there  is  also  a  greater  potential  for  bias.  To  identify  and  correct  for  bias,  it  will 
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be  important  to  consider  the  backgrounds  and  points  of  view  of  the  subjects  interviewed. 
All  things  considered,  it  is  important  to  note  that  the  researcher  is  not  immune  to  the 
same  bias  and  must  also  refrain  from  being  influenced  by  the  subjects  interviewed.  The 
researcher  must  endeavor  to  achieve  an  objective  assessment  of  the  data.  In  addition  to 
correcting  for  bias,  the  researcher  must  contend  with  interpreting  differing  interpretations 
provided  by  the  subjects  interviewed.  In  other  words,  there  is  the  potential  that  accounts 
perceived  to  be  different  are  in  actuality  converging  on  the  same  phenomenon.  That  said, 
sifting  through  interview  data  and  identifying  themes  from  the  conjecture  is  central  to 
forming  an  accurate  explanation  for  the  inconsistent  PRM  observed. 

To  effectively  and  thoroughly  search  for  relevant  themes  within  the  data,  the  data 
collected  from  interviews  must  be  as  complete  as  possible.  The  researcher  must  consider 
that  the  same  bias  proposed  as  a  contributor  to  inconsistent  PRM  also  has  the  potential  to 
take  affect  during  the  interview  process.  In  short,  the  subjects  being  interviewed  may  be 
inclined  to  skew  or  retain  data  that  would  adversely  affect  their  project,  reputation,  and/or 
organization.  In  anticipation  of  this,  the  subjects  slated  for  interview  will  be  informed 
that  all  identifiers  will  be  deleted  from  the  data,  and  pseudonyms  will  be  used.  In 
addition,  many  of  the  SMEs  available  to  the  researcher  for  in-person  interviews  will 
come  from  classified  programs  and  backgrounds  who  will  be  unable  to  divulge,  or  even 
acknowledge,  information  that  might  be  germane  to  the  research  study.  Although  the 
data  needed  is  expected  to  be  unrelated  to  classified  information,  subjects  will  be 
reminded  not  to  disclose  classified  information. 
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Implication 

If  executed  consistently,  PRM  has  the  potential  to  be  the  backbone  for  all  critical 
decision  making  across  the  Air  Force.  Short  of  full  potential,  the  author  has  made 
observations  working  directly  in  the  Air  Force  acquisitions  field  that  evidence  PRM 
deficiencies.  In  order  to  form  a  plan  for  improvement,  deficiencies  have  to  first  be 
identified  and  confirmed.  If  articulated  clearly  and  thoroughly  grounded  by  SMEs 
directly  in  the  field  of  study,  the  research  results  outlined  in  this  thesis  should  influence 
change.  Subsequently,  the  proposed  solution  and  recommendations  for  further  study  will 
be  a  starting  point  for  Air  Force  leadership  to  build  a  plan  for  improvement. 
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II.  Literature  Review 


Chapter  Overview 

To  develop  a  solution  that  requires  change,  it  is  crucial  to  first  identify  and 
understand  where  you  are  to  determine  how  to  get  to  where  you  want  to  be.  To  this  end, 
it  must  be  stated  that  the  author  has  observed  multiple  organizational  changes  within  the 
Air  Force  as  reactionary  and  aimless  instead  of  grounded  in  facts  and  due  process. 

Indeed,  creating  a  change  strategy  that  achieves  the  intended  effects  begins  with 
understanding  the  root  cause(s)  of  problems  with  the  current  process.  Working  directly 
in  the  field  of  study,  the  author  suspects  that  the  cause  for  inconsistent  PRM  across  the 
Air  Force  can  be  attributed  to  ambiguous  direction,  biases,  flawed  practices,  and 
inadequate  tools.  Further  investigation  must  be  done  to  sufficiently  explain  the  lack  of 
consistency  observed.  To  this  end,  literature  will  be  reviewed  to  gain  further  insight  to 
the  observations  made  and  to  discover  any  new  data  that  might  provide  further 
explanation.  Leveraging  valid  literature  germane  to  Air  Force  PRM  deficiencies  will  be 
critical  to  the  development  of  a  viable  solution  for  change. 

In  order  to  identify  what  drives  the  current  Air  Force  PRM  processes,  it  is 
necessary  to  consider  DoD  and  USAF  direction  on  the  matter  by  examining 
corresponding  publications.  This  is  a  natural  starting  point  since  compliance  with  such 
documentation  is  a  common  understanding  and  expectation  in  the  USAF.  That  said, 
attempts  at  compliance  with  USAF  documents  do  not  imply  the  publications  are 
necessarily  followed  or  they  can  be  intuitively  followed.  From  the  results  of  an  extensive 
study  that  assessed  the  PRM  practices  applied  to  127  various  projects,  Raz,  Shenhar,  & 
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Dvir  found  that  RM  practices  are  in  their  infancy  with  room  to  grow  (2002). 

Concurrently,  the  improvement  of  PRM  practices  is  contingent  on  the  improvement  of 
USAF  RM  publications.  However,  the  abundance  and  recent  proliferation  of 
publications  mandating  PRM  implementation  evidences  a  bona-fide  need.  As  an 
example,  a  company  grade  officer  (CGO)  in  the  USAF  is  subjected  to  the  direction  of 
seven  official  publications,  each  of  which  attempt  to  set  the  standard  for  risk  management 
practices  (AF  Policy  Directive  90-9,  AFI  90-901,  AFMC  Instruction  90-902,  AFMC 
Pamphlet  63-101,  DoD  5000. 2-R,  Air  Force  Pamphlet  63-128,  Air  Force  Instruction  90- 
802).  Local  policy  and  guidance  set  by  a  CGO’s  immediate  organization  and  different 
sources  of  training  and  education  encouraged  by  the  Air  Force  additionally  adds  to  the 
magnitude  of  guidance  to  be  followed. 

Beyond  USAF  publications,  the  literature  reveals  many  sources  which  address 
factors  that  influence  the  effectiveness  of  RM  to  achieve  the  best  course  of  action.  An 
increasing  number  of  case  studies  are  emerging  that  purport  RM  has  resulted  in  success 
and  a  lack  of  RM  has  led  to  poor  results.  Consequently,  the  lessons  learned  from  the  case 
studies  are  bolstering  support  and  growth  of  USAF  RM.  Yet  at  first  consideration,  the 
title  of  “risk  management”  could  lead  some  to  assume  a  narrow  scope.  On  the  contrary, 
an  increasing  number  of  organizations  are  realizing  the  value  of  RM  as  a  disciplined 
method  to  make  decisions  and  consider  opportunity  trade-offs.  The  increasing 
momentum  behind  USAF  RM  provides  an  opportunity  to  identify  causes  of  RM 
ineffectiveness  and  fine-tune  RM  processes. 
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Risk  Management  Concepts 

When  considering  only  the  literal  meaning  of  the  term,  risk  management,  it  could 
be  assumed  RM  is  a  practice  strictly  involving  the  management  of  a  “hazard,  chance  of 
bad  consequences,  loss,  exposure  to  chance  of  injury  or  loss”  (Concise  Oxford 
Dictionary,  201 1).  However,  further  investigation  reveals  a  broader  interpretation.  For 
instance,  the  UK  Association  for  Project  Management  (APM)  describes  risk,  as  it  applies 
to  RM,  as  “an  uncertain  event  or  set  of  circumstances  that,  should  it  occur,  will  have  an 
effect  on  the  achievement  of  the  project’s  objectives  (APM  Risk  Management  Specific 
Interest  Group  2010,  p.  16).”  Similarly,  the  US  Project  Management  Institute  (PMI) 
describes  RM  as  “an  uncertain  event  or  condition  that,  if  it  occurs,  has  a  positive  or 
negative  effect  on  a  project  objective”  (Nicholas  &  Steyn  2012,  p.  127).  At  a  basic  level, 
risk  management  definitions  upheld  by  their  respective  sources  as  cardinal  tend  to 
converge  on  a  singular  definition,  but  diverge  in  terms  of  process  and  technique.  For 
instance,  while  the  APM  and  PMI  definitions  give  leeway  to  account  for  opportunity,  Air 
Force  Pamphlet  (AFP)  63-128  only  considers  the  negative  aspect  of  RM  when  defining  a 
risk  as  “a  future  event  that,  if  it  occurs,  may  cause  a  negative  outcome  or  an  execution 
failure  in  a  program  within  defined  performance,  schedule,  and  cost  constraints” 
(Secretary  of  the  Air  Force  2014,  p.  84).  The  Space  and  Missile  Systems  Center  (SMC) 
Risk  Management  Process  Guide  cites  the  risk  definition  from  AFP  63-128  and 
additionally  encourages  consideration  of  potential  opportunities  but  does  not 
acknowledged  opportunity  as  the  reciprocal  of  RM  (Conrow,  2007).  Similarly, 
Kahneman  and  Tversky  (1979)  support  the  notion  that  consideration  of  opportunity  is  not 
necessarily  the  “dual,  mirror,  or  mirror  image  of  risk”  (p.  17). 
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Sources  consistently  describe  the  main  elements  of  RM  as  the  consideration  of  a 
future  event  that  has  a  potential  probability  and  impact.  The  Department  of  Defense  Risk 
Management  Guide  for  Defense  Acquisition  Programs  best  captures  the  most 
representative  definition  of  RM  as 

future  uncertainties  relating  to  achieving  program  technical  performance  goals 
within  defined  cost  and  schedule  constraints.  Defined  by  (1)  the  probability  of  an 
undesired  event  or  condition  and  (2)  the  consequences,  impact,  or  severity  of  the 
undesired  event,  were  it  to  occur  (Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering  2014,  p.  3). 

On  the  other  hand,  an  issue  or  crisis  refers  to  an  occurrence  in  the  present  tense  or  a  risk 
that  has  been  realized  (Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering, 
2014).  In  contrast,  the  AFP  63-128  stands  apart  from  the  rest  of  the  literature  as  it  does 
not  differentiate  between  an  issue  and  problem,  stating  that,  “an  issue  has  a  likelihood 
(probability)=l,  consequence  (impact)  >  0,  and  a  past,  present,  or  future  time  frame” 
(Secretary  of  the  Air  Force  2014,  p.  85). 

As  concepts  and  practices  move  beyond  the  basic  elements  and  become  more 
detailed,  it  is  necessary  to  specify  the  type  and  application  of  the  risk  being  referenced. 
This  specification  is  necessary  because  there  exist  several  types  of  RM  based  on 
application.  The  Defense  Systems  Management  College1  (DSMC)  RM  Guide  for  DoD 
Acquisitions  observed  that,  “the  Services  differ  in  their  approaches  to  Risk  Management. 
Each  approach  has  its  strengths  but  no  one  approach  is  comprehensive”  (Defense 
Acquisitions  University  2001,  p.  3).  In  regards  to  Air  Force  risk  management,  there  are 

1  Co-located  with  DAU  Headquarters  at  Fort  Belvoir,  Virginia,  the  Defense  Systems  Management  College 
(DSMC)  is  chartered  to  provide  education  for  the  DoD  Acquisition,  Technology  &  Logistics  (AT&L) 
workforce  across  the  globe. 
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three  distinct  classifications  observed  within  the  extant  literature:  Health  (ESOH)/Air 
Force  Occupational  Safety  &  Health  (AFOSH),  Operational  Risk  Management  (ORM), 
and  Project  Risk  Management  (PRM).  Furthermore,  Air  Force  PRM  can  be  considered  a 
level  that  is  intended  to  populate  the  DoD  enterprise  or  macro  risk  management  level. 

The  literature  does  not  specifically  differentiate  between  a  distinct  project  PRM  level  and 
an  Enterprise  RM  level;  however,  a  hierarchal  existence  of  these  levels  is  implied.  As 
such,  the  focus  of  this  thesis  will  be  on  Air  Force  PRM  and  its  impact  on  DoD  enterprise 
risk  management. 

Air  Force  PRM  applies  to  the  cradle-to-grave  development  of  a  project.  The 
Challenger  disaster  drove  rigor  in  process  based  risk  analysis  which  began  to  emerge 
formally  in  the  early  1990’s  (Altabbakh,  Murray,  Grantham,  &  Damle,  2013).  The 
majority  of  AF  publications  on  the  topic  of  PRM  are  consistent  in  describing  an  iterative 
RM  process.  Specifically,  DoD  5000. 2R  captures  the  most  representative  list  of  the  RM 
process  steps  in  order  as  “planning,  assessment  (identification  and  analysis),  handling, 
and  monitoring”  (Under  Secretary  of  Defense  2001,  p.  71).  It  is  common  to  encounter 
terminology  inconsistencies  between  publications,  and  it  is  the  case  that  these 
inconsistencies  ultimately  lead  to  office  debates.  The  DoD  5000. 2R  attempts  to  account 
for  this  by  noting  how  risk  identification  and  analysis  should  be  included  in  the 
assessment  phase.  Indeed,  Air  Force  Pamphlet  (AFP)  63-128  identifies  the  same  process 
as  DoD  5000. 2R  but  clarifies  ambiguity  surrounding  the  risk-handling  step  when  stating 
that 

the  Fife  Cycle  Risk  Management  (FCRM)  process  model  expands  the  title  of  this 
step  from  “Risk  Mitigation  Planning”  in  the  Risk  Management  Guide  for  DoD 
Acquisition  to  “Risk  Handling  Planning  and  Implementation”  to  recognize  that 
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most  of  these  options  address  handling  risk  in  a  manner  other  than  mitigating  (i.e. 

eliminating  or  reducing)  it  (Secretary  of  the  Air  Force  2014,  84). 

The  Space  and  Missile  Center  (SMC)  Risk  Management  Process  Guide  also  aligns  with 
AFP  63-128  by  directly  citing  the  pamphlet  for  its  own  definition  (Conrow  2014,  p.  10). 
The  educational  textbook  titled  Project  Management  for  Engineering,  Business  and 
Technology  from  PMI’s  Project  Management  Body  of  Knowledge  (PMBOK),  a 
benchmark  for  the  DoD  and  industry,  describes  a  very  different  version  of  the  RM 
process  as,  “identify,  assess,  plan  risk  response,  and  track  and  control”  (Nicholas  & 
Steyn,  2012). 

Within  the  LCRM  process,  the  risk-handling  step  provides  an  opportunity  to  take 
action  towards  limiting  the  effects  of  the  identified  risk.  The  literature  consistently 
describes  risk-handling  as  a  set  of  options,  yet  the  literature  differs  on  the  options  listed. 
One  representation  of  the  majority  of  the  literature  regarding  risk  handling  comes  from 
the  Air  Force  Pamphlet  63-128  which  lists  the  risk  handling  options  as  “accepting, 
monitoring,  transferring,  mitigating  (or  controlling),  and  avoiding”  (Secretary  of  the  Air 
Force  2014,  p.  84).  The  pamphlet  clarifies  a  common  point  of  ambiguity  in  terminology 
by  identifying  that  controlling  a  risk  is  synonymous  with  mitigating  a  risk.  In  contrast, 
AFMC  Pamphlet  63-101  lists  changing  the  program  as  an  option  that  is  preferable  to  risk 
avoidance,  risk  acceptance  and  risk  assumption  (Sawdy,  1997).  On  the  other  hand,  PMI 
PMBOK  encourages  risk  reduction  instead  of  risk  mitigation  and  adds  contingency 
planning  as  an  option  (Nicholas  &  Steyn,  2012).  The  inconsistencies  between  said 
publications  lead  to  ambiguity  and  confusion  for  those  responsible  for  project  risk 
management  in  the  United  States  Air  Force. 
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Evolving  Need  for  Air  Force  Risk  Management 

To  a  degree,  the  claim  could  be  made  that  managing  risk  is  already  an  intuitive 
part  of  life.  RM  is  implemented  several  times  a  day  without  thinking  of  it  as  a  formal 
process  (Ourada,  2013).  Haussermann  (2006)  adds  that  “every  time  we  cook  a  meal, 
walk  down  the  street,  ride  a  bike,  drive  our  car,  participate  in  sports,  watch  television, 
purchase  a  product  or  decide  where  to  live  we  are  choosing  the  best  course  of  action  for  a 
specific  given  situation”  (p.  7).  For  these  simple  events,  a  drawn-out  process  to  conclude 
the  proper  course  of  action  is  not  necessary.  However,  as  the  Air  Force  evolves  and 
becomes  more  complex,  the  best  decisions  about  the  future  become  less  intuitive. 

PMI PMBOK  lists  the  causes  for  higher  project  risk  complexity  as:  uniqueness; 
team  inexperience;  ambiguous  concept  of  risk  and  unknowns;  use  of  an  unusual 
approach;  developing  a  system  while  furthering  technology  at  the  same  time;  developing 
and  testing  new  equipment  systems,  or  procedures;  and  operating  in  an  unpredictable  or 
variable  environment  (Nicholas  &  Steyn,  2012).  It  is  the  author’s  experience  that  these 
characteristics  are  indicative  of  Air  Force  acquisitions  development  and  necessitates  bon¬ 
a-fide  PRM  process  rigor.  Evidence  in  alignment  with  this  position  can  be  found  in  a 
white  paper  authored  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  in  2013  entitled  Better  Buying  Power  3.0.  This  paper  emphasized  the 
priority  to  improve  leaders’  ability  to  understand  and  mitigate  project  risk.  Specifically, 
the  Under  Secretary  of  Defense  states,  “successful  product  development  requires 
understanding  and  actively  managing  program  risks.  Acquisition  professionals  may 
debate  the  best  approach  for  managing  risk,  but  they  agree  that  effective  qualitative  and 
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quantitative  risk,  issue,  and  opportunity  management  are  critical  to  a  program’s  success” 
(Kendall  2013,  p.  31). 

There  are  also  case  studies  revealing  a  trend  that  lack  of  risk  management  often 
leads  to  disaster.  One  of  the  most  notable  cases  as  it  relates  to  space  operations  and  the 
Air  Force  is  the  January  1986  explosion  of  the  Space  Shuttle  Challenger  73  seconds  after 
liftoff  (Altabbakh  et  al.,  2013).  Though  unfortunate,  these  events  drove  the  adherence 
and  refinement  of  PRM.  With  growing  acceptance  of  RM  as  a  value  added  discipline, 
RM  has  emerged  as  a  popular  subject  area  in  acquisitions  academia.  Various  recognized 
institutions  and  RM  experts  introduced  forms  of  project  risk  management:  examples 
include  the  work  of  Boehm  (1991),  Fairley  (1994),  Dorofee  et  al.  (1996),  the  Project 
Management  Institute  (PMI,  1996),  Kliem  and  Ludin  (1997)  and  Chapman  and  Ward 
(1997).  One  notable  example  is  found  in  the  work  of  Couillard  (1995)  who  investigated 
different  project  management  approaches  with  the  objective  of  reducing  the  influence  of 
risk  and  increasing  the  likelihood  of  project  success.  Ward  (1999)  also  studied  how 
project  context  and  participant  characteristics  influence  implementation  of  project  risk 
management  processes.  Shenhar  and  Dvir  (1996)  proposed  a  typological  theory  of 
projects  based  on  two  dimensions:  technological  uncertainty  and  system  scope  (or 
complexity).  These  efforts  to  formalize  and  refine  RM  practices  by  the  aforementioned 
authors  are  a  sign  of  improvement  as  well  as  necessary  first  steps.  However,  the  stove- 
piped  creation  of  direction  and  guidance  by  various  entities  has  led  to  inconsistency  even 
though  improvement  has  and  continues  to  be  made.  That  said,  instruction  and  guidance 
available  to  Air  Force  Company  Grade  Officers  is  currently  ambiguous  and  fragmented. 
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Air  Force  Project  Risk  Management  Guidance  and  Directives 

As  a  company  grade  officer  in  the  United  States  Air  Force,  this  author  has  been 
required  to  provide  risk  management  on  numerous  projects  and  has  wrestled  with 
identifying  a  comprehensive  location  and  source  for  risk  management  guidance  and 
instruction.  The  Defense  Acquisition  University  community  of  practice  (CoP)“  strives  to 
“provide  the  Risk  Community  with  a  Risk  Management-focused  website  and  companion- 
linked  DoD/DAU  Risk  Guidebook  that  is  the  first  source  to  which  a  novice  or 
experienced  practitioner  will  go  to  get  answers  and  obtain  a  pulse  on  the  state  of  Risk 
Management”  (COP  Risk  Management  Section  Editor;  need  CAC  to  find).  The  CoP 
makes  a  valid  attempt  to  consolidate  RM  data,  but  only  succeeds  in  reiterating  ambiguity 
created  by  redundant  DoD  and  USAF  publications.  Another  website,  touted  just  as 
official  as  the  Defense  Acquisition  University’s  COP  website,  is  Acqnotes  .  AcqNotes 
attempts  to  categorize  the  relative  DoD  Risk  Management  documentation  into  categories 
of  instruction,  manuals,  directives,  and  guidance,  but  does  not  specify  which  particular 
set  of  documents  trumps  which  or  is  most  relevant  where  overlap  occurs. 

The  DSMC  sponsored  a  working-group  that  investigated  RM  Policy  in  the  DoD 
and  determined  that  RM  policy  contained  in  the  DoDD  5000.1  is  “not  comprehensive  and 
structured  approaches,  when  they  exist,  are  similar  to  the  DoD’s  approach  to  Risk 
Management”  (Defense  Acquisition  University  2001,  p.  3).  The  working-group  also 
recommended  that  “techniques  in  the  DoD  5000. 2-R  be  expanded,  using  the  Defense 
Acquisition  Desk-book  as  the  expansion  means,  in  order  to  provide  comprehensive 

2  This  document  can  be  retrieved  from  https://acc.dau.mil/communitybrowser.aspx. 

3  This  document  can  be  retrieved  from  http://acqnotes.com. 
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guidance  for  the  implementation  of  risk  management  policy”  (Acquisition  University 
2001,  p.  3).  The  Defense  Acquisition  Guidebook  (DAG),  also  referred  to  as  the  Desk- 
book,  is  consistently  recommended  as  the  primary  guide  for  DoD  acquisitions  and 
addresses  what  DoD  Risk  Management  is  and  how  to  execute  it,  as  well  as  an  overview 
on  multiple  other  sets  of  instruction,  manuals,  directives,  and  guidance  publications. 
Contrarily,  the  Department  of  Defense  Risk  Management  Guide  for  Defense  Acquisition 
Programs  claims  that  “program  Managers  (PM)  are  responsible  for  managing  risk  in 
accordance  with  the  mandatory  requirements  contained  in  the  DoD  Instruction  (DoDI) 
5000.02,  “Operation  of  the  Defense  Acquisition  System,”  and  are  required  to  outline  their 
risk  management  strategy  in  accordance  with  the  Systems  Engineering  Plan  (SEP) 

Outline  (2011)”  (Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  2014, 
p.  1).  Regardless  of  how  one  interprets  the  recommendations  and  mandates  within  the 
literature,  the  number  of  RM  frameworks  developed  contributes  to  an  overall  uncertainty 
regarding  the  essential  components  of  RM  (Lundqvist,  2014). 

Although  not  required,  DoD  and  USAF  publications  consistently  recommend  the 
development  of  a  project  or  unit  level  Systems  Engineering  Plan  (SEP)  with  a  section 
dedicated  to  RM  that  defers  greater  detail  to  a  separate  risk  management  plan  (RMP). 

The  SEP  and  RMP  provide  an  opportunity  to  interpret  the  convoluted  RM  publications 
and  tailor  a  RM  approach  that  is  unique  to  the  organization.  Implementing  a  SEP  and 
RMP  also  lends  itself  to  a  definitive  process  and  consistency  at  the  project  or  unit  level. 
However,  because  the  unit  level  RM  documents  are  based  on  the  ambiguous  higher-level 
Air  Force  publications,  there  is  inconsistency  between  organizations.  The  2012 
Acquisition,  Technology,  and  Logistics  article  by  Mercado,  Schwartz,  and  Ivie,  supports 
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this  view  with  the  observation  that,  in  regards  to  AF  RM,  “process  inconsistencies  were 
leading  to  inefficiency  and  confusion”  across  the  board  (p.  22).  Bolstering  the  need  for 
consistency,  Haussermann  (2006)  makes  the  point  in  his  article  that  standardization  is  the 
most  necessary  requirement  for  a  successful  risk  management  plan.  There  are  two 
consequences  that  result  from  this  inter-organizational  inconsistency.  First,  active  duty 
Air  Force  CGOs  change  organizations  as  often  as  every  two  years  or  at  least  every  four 
years.  Considering  the  turnover  rate,  it  is  inefficient  for  CGOs  to  relearn,  just  as  it  is 
inefficient  for  the  organization  to  reteach  unique  RM  processes  at  such  a  frequent  rate. 
Along  these  same  lines,  changes  will  be  induced  upon  the  organization  receiving  the 
frequent  influx  of  new  CGOs  who  will  attempt  to  impose  their  previously  learned  styles 
of  RM.  Second,  macro  level  risk  management,  specifically  referred  to  as  enterprise  risk 
management  (ERM),  will  be  based  on  inconsistent  comparisons  between  units,  which  can 
lead  to  erroneous  decision-making.  Based  on  these  facts,  it  can  be  argued  that  the 
effectiveness  of  enterprise  risk  management  relies  on  consistent  unit  level  PRM  where 
consistency  is  sensible  and  necessary.  In  other  words,  higher-level  PRM  guidance  must 
find  a  balance  between  mandating  the  correct  amount  of  restrictiveness  to  promote 
enterprise  level  PRM  consistency  without  restricting  units  from  tailoring  their  unit-level 
PRM  plans  to  accommodate  unique  missions  and  operations. 

Risk  Assessment  Methods 

Accuracy  in  risk  assessment  (i.e.,  identification  and  analysis)  is  the  bedrock  to  not 
only  achieving  RM  consistency  between  organizations,  but  also  to  carry  out  an  effective 
RMP  within  an  organization.  Ideally,  using  one  method  for  risk  assessments  would 
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contribute  to  consistency;  however,  this  approach  would  be  unrealistic  considering  that 
every  mission  and  effort  is  unique.  For  this  reason,  there  are  a  multitude  of  techniques 
prescribed  for  AF  project  RM  in  the  literature  ranging  from  quantitative  to  qualitative  and 
from  simple  to  complex.  As  an  example,  Gilb  (2002)  outlines  a  variety  of  risk 
assessment  tools  that  lend  themselves  to  complex  projects,  although  the  methods 
described  by  Snyder  and  Buede  (2000)  are  more  suitable  for  smaller  projects.  PMI’s 
PMBOK  provides  a  more  encompassing  list  of  nine  different  approaches  by  which  to 
assess  project  risk  succeeding  in  covering  all  approaches  mentioned  elsewhere  in  the 
literature  as  well  as  providing  the  most  thorough  instruction  (Nicholas  &  Steyn,  2012). 

Despite  the  number  of  options  proposed,  the  author  has  only  observed  one  risk 
assessment  method  that  is  consistently  used  in  the  field;  this  method  is  commonly 
referred  to  as  a  risk-cube  or  risk-matrix.  Siefert  (2007)  describes  that  among  the  multiple 
ways  developed  to  track  and  manage  risks,  “perhaps  the  most  common  method  is  the  use 
of  the  ubiquitous,  basic  “Risk  Cube”  (p.  18).  It  is  possible  that  the  risk  cube  approach  is 
common  because  it  intuitively  assesses  the  severity  of  a  risk  on  the  basis  of  probability 
and  impact  between  an  X  and  Y-axis.  The  NASA  Systems  Engineering  Handbook 
defines  this  approach  similarly  as  the  “5x5  risk  matrix  methodology,”  which  they  go  on 
to  describe  as  a  sufficient  way  to  “combine  qualitative  and  semi-quantitative  measures  of 
likelihood  with  similar  measures  of  consequences”  (NASA,  2007,  p.  46).  However,  the 
NASA  handbook  also  warns  that  there  are  several  drawbacks,  including  “the  fact  that 
combinatorial  interactions  among  multiple  risks  are  not  accounted  for,  the  impact  of 
aggregate  risks  is  not  addressed,  uncertainty  quantification  is  lacking,  and  general  over¬ 
simplicity,”  concluding  that  the  method,  “falls  short  in  providing  context  behind  each 
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risk”  (NASA,  2007,  p.  52).  History  has  shown  the  consequences  of  overlooking  the 
aggregate  impact  of  smaller  risks.  For  instance,  the  chemical  plant  accident  at  Bhopal, 
India  was  due  to  30  separate  risks,  and  the  Chernobyl  disaster  was  due  to  6  errors  that 
when  considered  individually  were  trivial  and  ignored,  but  jointly  were  high  impact 
(Nicholas  &  Steyn,  2012).  In  his  article  Transforming  Project  Risk  Management  into 
Project  Uncertainty  Management ,  Chapman  (2001)  also  criticizes  that  the  over 
simplification  of  risk  matrix  estimates  of  impact  and  associated  probability  results  in 
unnecessary  uncertainty.  In  alignment  with  Chapman,  Cioaca  (201 1)  further  warns  that 
true  interpretation  of  comparisons  in  a  risk  matrix  must  take  into  consideration  the  risk 
attitude  and  subjective  judgments  used  by  those  who  constructed  it.  Simply  stated,  inputs 
to  risk  matrices  and  resulting  outputs  require  subjective  interpretation.  Resultantly, 
Cioaca  (2011)  suggests,  “that  risk  matrices  should  be  used  with  caution  and  only  with 
careful  explanations  of  embedded  judgment”  (p.  81-82).  Although  the  risk  cube  is 
popular  and  commonly  considered  best  practice,  “there  has  been  very  little  empirical 
evidence  from  studies  showing  its  actual  efficacy  in  managing  risks”  (Cox,  2008,  p.  497). 
The  DSMC  Risk  Management  Guide  for  DoD  Acquisition  (Fourth  Edition)  suggests  that 
instead  of  faulting  the  risk-cube  method,  individuals  should  consider  that  in  most  cases, 
“risk  scales  are  actually  just  raw  (uncalibrated)  ordinal  scales,  reflecting  only  relative 
standing  between  scale  levels  and  not  actual  numerical  differences”  (Defense  Acquisition 
University  2001,  p.  24).  Though  it  may  be  convenient  and  simple  to  blame  a  method  or 
tool  for  inconsistency,  and  though  the  method  or  tool  used  may  be  a  contributor,  there 
exist  a  deeper  root  cause.  This  is  because  the  very  essence  of  risk  management  is  based 
on  predicting  the  future  (which  is  always  difficult  at  best)  and  drawing  on  assumptions 
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that  often  leave  vast  opportunity  for  subjectivity.  Indeed,  qualitative  methods  breed 
subjective  inputs  and  can  hardly  be  avoided,  even  on  the  most  technical  efforts.  On  the 
basis  of  this,  it  may  be  the  case  that  the  risk-cube  exacerbates  the  inherent  subjectivity  of 
RM. 

Altabbakh  et  al.  (2013)  would  argue  that  risk  matrices  are  not  to  blame  for  risk 
assessment  error,  as  “engineering  managers  should  note  that  there  is  no  one  single  perfect 
model  for  risk  assessment”  (p.  24).  Rather,  whenever  there  is  risk  and  uncertainty,  there 
is  subjectivity.  Cargill  and  Moore  (2013)  also  support  this  line  of  thought  when  noting 
that 

Bias  gets  introduced  not  through  flaws  in  any  of  the  methods  but  through  bias  in 
the  assumptions  and  factors  used  in  the  analysis.  Thus,  we  need  to  delve  more 
deeply  into  opinion,  bias,  and  subjectivity,  and  how  these  factors  influence  our 
risk  and  uncertainty  assessments.  The  pertinent  question  becomes,  what  do  we 
need  to  address  this  subjectivity?  (p.  30) 

Subjectivity  gives  leeway  to  bias,  and  a  majority  of  the  literature  reviewed  recognizes 
bias  as  the  root  cause  of  risk  assessment  error,  and  by  extension,  risk  management 
ineffectiveness.  Ultimately,  biases  can  lead  to  undesired  consequences,  including 
inaccurate  project  estimates  (Flyvbjerg,  2006).  Specifically,  the  author  has  identified 
twenty-two  notable  biases  outlined  by  the  literature  and  has  sought  to  provide  clarity 
through  categorization  based  on  cause  and  effect.  Among  the  twenty-two  notable  biases, 
two  of  the  most  important  categories  based  on  cause  were  cognitive  and  incentive  bias 
and  two  of  the  most  important  categories  based  on  effect  were  optimism  and  pessimism 
bias. 
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Cognitive  Effects  on  Risk  Management 

There  is  a  natural  underlying  apprehension  to  risk  management.  Cargill  and 
Moore  (2013)  contend  that,  “under  conditions  of  commitment  and  optimism  for  the 
future,  no  one  likes  to  recognize  risk”  (p.  31).  General  reluctance  towards  PRM  can  be 
only  partially  overcome  by  mandating  or  forcing  the  endeavor.  PRM  data  and  actions 
will  still  occur,  but  a  lack  of  intrinsic  motivation  will  ultimately  result  in  a  lack  of 
effectiveness  throughout  the  process;  a  phenomenon  often  referred  to  as,  ‘trash  in,  trash 
out.’  Behaviorally,  the  author  has  observed  a  resulting  bias  to  underestimate  likelihood 
and/or  impact  of  risk  yielding  in  overly  optimistic  assessments.  For  instance,  Cargill  and 
Moore  (2013)  point  out  that,  “there  is  a  natural  tendency  to  be  aggressive  with 
assumptions  early  in  a  program  to  make  the  program  appear  attractive”  (p.  30).  The 
literature  dovetails  with  the  author’s  career  observations  that  those  qualified  to  assess  risk 
are  also  team  members  whose  interests  are  aligned  with  projecting  confidence  in  project 
success.  Presenting  an  assessment  that  conveys  that  the  project  is  in  trouble  can  lead  to 
cancellation  of  funds  or  even  a  cancellation  of  the  project  itself.  At  a  minimum,  it  will 
bring  unwanted  attention  in  the  form  management  oversight  and/or  reporting  which 
further  taxes  the  team’s  resources  and  energy  that  should  instead  be  devoted  to  the  actual 
risks.  It  would  make  sense  to  assume  that  high-risk  attention  would  result  in  aid  in  the 
form  of  additional  resources  and  support;  however,  it  is  the  author’s  experience  that  there 
is  little  that  can  be  done  by  external  audiences  once  the  high  risk  has  been  identified  other 
than  exacerbate  the  risk  itself.  Baram  (2003)  captures  this  phenomenon  as  impedance  to 
good  risk  assessment  due  to  “conflict  of  interest  and  bias  by  project  managers  toward 
reporting  good  news”  (p.  2).  Similarly,  many  engineers  do  not  want  to  admit  there  are 
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any  risks  associated  with  the  project,  and  many  program  managers  do  not  want  to  present 
negative  aspects  of  their  program  to  higher  management  or  the  customer  (Kasperson  & 
Kasperson,  1996).  As  a  result,  incentive  biases  to  avoid  the  pressure  and  interference 
from  external  audiences  strategically  results  in  an  overly  optimistic  risk  report  and 
ultimately  transcends  to  mishandling  risk. 

A  key  bias  that  has  proven  detrimental  is  the  failure  to  recognize  and  mitigate 
risks  that  have  a  very  high  impact  and  a  very  low  probability  of  occurrence.  Pate-Comell 
(2015)  acknowledge  the  oversight  of  this  type  of  risk  and  refer  to  it  as  a  black  swan  or 
perfect  storm.  Pate-Cornell  (2015)  also  describes  these  risks  using  the  same  terminology 
and  explains  that  they  are  “sometimes  dismissed  after  the  fact  as  having  been 
unimaginable  before  they  happened”  (p.  65).  There  is  a  cognitive  tendency  to 
subconsciously  dismiss  these  risks  due  to  their  low  probability  and  a  conscious  incentive 
to  avoid  expending  precious  resources  for  the  same  reason.  In  fact.  Trope,  Liberman,  and 
Wakslack  (2007)  use  the  term  optimism  bias  to  describe  this  phenomenon  and  contend 
that  it  is  evidenced  across  a  broad  range  of  different  contexts.  For  example,  people  tend 
to  underestimate  their  likelihood  of  having  a  heart  attack  (Avis,  Smith,  &  McKinlay, 
1989),  being  involved  in  a  car  accident  (McKenna,  Stainer,  &  Lewis,  1991),  being  the 
victim  of  crime  (Perloff  &  Fetzer,  1986),  and  having  an  unwanted  pregnancy  (Burger  & 
Bums,  1988).  Investigations  revealed  that  the  siting  of  the  Fukushima  Daiichi  reactors 
and  a  5.7  magnitude  tsunami  design  criterion  was  chosen  despite  the  knowledge  that 
earthquakes  of  magnitude  greater  than  8  (and  the  large  tidal  waves  that  came  with  them) 
had  occurred  several  times  in  recorded  history  (Pate-Comell  &  Cox,  2014).  It  could  be 
argued  there  was  another  related  bias  in  effect  known  as  a  disproportionate  weighting  of 
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recent  events,  or  in  the  case  of  the  reactors,  a  recent  lack  of  earthquake  activity  (Protiviti, 
2014).  The  biases  mentioned,  which  include  weighting  of  recent  events,  optimism  bias, 
and  failure  to  acknowledge  the  probability  of  a  “perfect  storm”  or  “black  swan”  are 
related  as  errors  in  assessing  the  evidence  at  hand. 

Similar  to  the  concept  of  optimism  bias  is  a  phenomenon  known  as  the  planning 
fallacy,  which  refers  to  overly  optimistic  performance  of  the  project  management  team 
(Buehler,  Griffin,  &  Ross,  1994).  The  planning  fallacy  involves  the  genuine  belief  that  a 
project  or  effort  will  not  take  as  long,  cost  as  much,  and/or  will  produce  a  better  product 
than  what  actually  comes  to  fruition.  The  root  cause  of  the  resulting  oversight  is  the 
underestimation  of  risk  factors  that  trigger  the  poor  performance.  It  can  be  deduced  from 
these  facts  that  planning  fallacies  contribute  to  a  commonly  stated  problem  throughout 
the  space  system  design  community  that  severe  deficiencies  exist  within  system  cost  and 
schedule  estimating  methodologies  (Reeves,  Eveleigh,  Holzer  &  Sarkani,  2013). 

Cognitive  bias,  in  contrast  with  incentive  based  conscious  bias,  occurs  deeper  in 
the  subconscious  where  it  can  take  effect  unrealized  and  is  therefore  harder  to  detect  and 
find  fault.  The  literature  most  commonly  attributed  cognitive  error  to  groupthink,  which 
is  defined  as  a  pattern  of  thought  characterized  by  self-deception,  forced  manufacture  of 
consent,  and  conformity  to  group  values  and  ethics  (Merriam-Webster,  2008).  The  Air 
Force  Risk  Identification,  Integration  and  “illities”  Guide  notes  that  with  respect  to 
groupthink,  the  “literature  from  the  field  of  cognitive  psychology  generally  suggests  that 
people  often  have  difficulty  in  characterizing  the  relative  risks  of  various  activities 
appropriately  (possibly  due  to  ‘group-think’),  thereby  resulting  in  underestimation  of 
their  effects”  (2008,  p.  5).  Protiviti  (2014)  more  specifically  identifies  contributors  to 
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groupthink  such  as  “not-invented-here  bias”  and  “confirmation  bias.”  Though  not 
necessarily  espoused  as  often,  many  other  sources  framed  groupthink  as  the  end  result  of 
the  many  other  cognitive  biases.  This  is  logical  considering  that  a  risk  management  team 
would  tend  to  experience  the  same  cognitive  bias  factors  and,  thus,  be  persuaded  as  a 
group  thereby  triggering  groupthink  as,  “conformity  to  group  values  and  ethics” 
(Merriam- W  eb  ster) . 

Groupthink  is  not  reported  in  the  literature  as  definitively  favoring  an 
underestimation  or  overestimation  of  risk.  However,  many  of  the  cognitive  bias  factors 
that  lead  to  groupthink  tend  to  either  lead  to  an  underestimation  or  overestimation  of  risk. 
For  instance,  simply  adding  more  detail  to  a  risk  description  can  make  the  scenario  more 
vivid  and  thus  more  likely  when,  on  the  contrary,  knowing  more  detail  lowers  its 
probability  by  definition  (Pate-Comell,  2015).  This  is  reminiscent  of  the  affect  heuristic; 
that  is  to  say,  absence  of  proof  is  not  proof  of  absence.  Pate-Comell  (2015)  and  Protiviti 
(2014)  also  identify  a  cognitive  bias  phenomenon,  termed  as  confirmation  bias,  where 
evidence  is  truncated  or  interpreted  in  a  way  that  confirms  one's  preconceptions.  Per 
investigations,  confirmation  bias  was  one  of  the  root  causes  for  the  2011  accident  in  the 
Fukushima  Daiichi  nuclear  reactor  disaster  (Pate-Cornell,  2015).  Both  Pate-Comell 
(2015)  and  Cargill  and  Moore  (2013)  agree  that  anchoring  belief  bias  has  proven 
detrimental  as  a  subset  of  confirmation  bias.  Pate-Cornell  (2015)  identifies  the  broadest 
range  of  biases  amongst  the  literature  to  include  not-invented-here  bias,  dominate 
personalities,  overreliance  on  numbers,  framing  effect,  availability  heuristic,  hindsight, 
the  ostrich  effect,  getting  along  and  disregard  for  contrary  information.  Pate-Cornel 
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(2015)  effectively  provides  evidence  and  an  understanding  for  the  chaos  that  cognitive 
biases  induce  on  risk  management. 

A  bias  that  is  always  imminent  is  the  potential  to  erroneously  skew  risk  handling 
toward  doing  too  much  or  too  little.  Too  much  risk  handling  is  often  sought  after  as  the 
optimal  goal  but  it  does  not  come  without  the  expense  of  precious  resources  and  time. 
Being  too  conservative  with  risk  handling  as  a  way  to  conserve  precious  resources  and 
time  can  leave  a  project  facing  the  full  brunt  of  accepting  the  risks,  while  doing  too  much 
can  overtax  available  resources.  The  literature  alludes  to  finding  the  appropriately 
balanced  handling  strategy  unique  to  each  risk  by  recommending  multiple  handling 
methods;  accepting,  monitoring,  transferring,  mitigating  (or  controlling),  and  avoiding 
(Secretary  of  the  Air  Force  2014,  p.  84).  The  Department  of  Defense  Regulation 
5000.2R  directs  that  the  risk  handling  “strategy  shall  explain  how  the  risk  management 
effort  shall  reduce  system-level  risk  to  acceptable  levels”  (p.  27).  Nicholas  and  Steyn 
(2012)  provide  an  alternate  perspective  where  risk  should  be  accepted  only  when 
avoiding,  reducing,  or  transferring  exceeds  the  benefits. 

Failure  to  find  this  balance  due  to  an  error  in  judgment  is  minuscule  compared  to 
a  group,  or  in  many  cases,  an  entire  organizational  culture  that  is  skewed  towards  either 
risk  avoidance  or  risk-taking  (Protiviti,  2014).  Although  the  U.S.  Air  Force  is  very  large, 
and  not  every  unit  and  mission  is  the  same,  the  literature  alludes  to  an  overall  risk  averse 
culture.  For  example,  Greiner,  Dooley,  Shunk,  and  McNutt  (2002)  launched  a 
comprehensive  study  that  conducted  interviews  with  key  members  of  the  multi-functional 
Air  Force  Corporate  Structure  (AFCS)  charged  with  making  decisions  regarding  resource 
allocation  against  all  Air  Force  activities,  including  weapon  systems  acquisition 
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programs.  In  response  to  an  open-ended  question  regarding  flaws,  AFCS  representatives 
interviewed,  28  percent  described  a  system  that  seems  to  be  risk  averse  in  nature.  This 
finding  suggests  that  the  other  72  percent  of  those  interviewed  would  consider  the  system 
to  not  be  risk  adverse  in  nature. 

Although  trying  to  eliminate  risk  seems  like  a  noble  cause,  “the  prime  symptom 
of  “trying  to  eliminate  risk”  is  micromanagement:  excessive  controls,  unrealistic 
documentation  requirements,  and  trivial  demands  for  the  authorization  of  everything 
which  can  stifle  innovation”  (Nicholas  &  Steyn,  4th  ed  2012,  p.  374).  Per  Aronstein  and 
Piccirillo  (2012),  being  overly  risk  averse  “forces  a  company  into  a  plodding,  brute  force 
approach  to  technology,  which  can  be  far  more  costly  in  the  long  run  than  a  more 
adventurous  approach  where  some  programs  fail  but  others  make  significant  leaps 
forward”  (p.  79-80).  The  author  has  experienced  a  risk  averse  culture  in  the  Air  Force 
Space  program.  Due  to  the  nature  of  space  access,  risk  management  termed  mission 
assurance  in  the  USAF  space  community  should  have  a  lower  than  average  tolerance  for 
risk.  However,  the  notion  of  mission  assurance  was  taken  to  such  an  extreme  by  upper 
management  that  mission  cost  and  schedule  overruns  were  the  only  outcome  assured. 
Project  team  members,  having  direct  involvement  with  the  effort,  were  often  best  suited 
to  make  risk  management  trade-off  decisions  but  were  often  overridden  by  senior 
management.  Thus,  a  major  challenge  for  an  organization  implementing  ERM  is  to 
ensure  that  senior  management  decision-making  is  contingent  on  business  managers 
throughout  the  firm  and  takes  proper  account  of  the  risk-return  tradeoffs  (Stulz,  2006). 
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Risk  Management  Process  Solutions 

As  identified  previously,  there  is  disagreement  between  sources  designated  as 
official  Air  Force  RM  publications.  There  exist  websites  that  attempt  to  centralize  and 
organize  these  documents,  which  admittedly  adds  convenience  for  accessing  PRM 
resources,  but  neither  delineates  applicability  or  precedence  of  overlapping  publications. 
The  contradiction  found  in  official  Air  Force  PRM  publications  is  counterproductive  to 
process  consistency.  Furthermore,  the  number  of  risk  management  publications  in  other 
military  branches  found  in  contradiction  provides  an  analogous  inconsistency  at  the 
enterprise  level  of  PRM.  The  solution  may  be  to  comply  with  Air  Force  Instruction  33- 
360  (2013),  which  states  that  “all  publications  must  be  at  least  as  restrictive  as  the  higher 
headquarters  publication  they  implement  and  must  not  contradict  the  higher  headquarters 
publication”  (p.  42).  Subsequently,  the  next  step  would  be  to  require  industries  under 
government  contract  to  report  risks  consistent  with  government  assessment  measures. 
Mattice  (2013)  explains  that,  “industry  shouldn’t  feel  like  the  forgotten  stepchild, 
however,  as  government  agencies  are  themselves  highly  fragmented  into  what  many  refer 
to  as  “silos  of  excellence”  “  (p.  22).  It  is  the  author’s  experience  that  most  companies 
under  a  contract  with  the  government  implement  their  own  homegrown  risk  management 
processes  as  opposed  to  attempting  to  decipher  DoD  instruction  and  parallel  their 
practices  with  the  Air  Force.  This  may  be  effective  for  the  company  and  the  effort  they 
have  been  contracted  for,  but  it  adds  yet  another  layer  of  inconsistency  when  consolidated 
and  compared  at  the  enterprise  level.  Synchronizing  government  risk  management 
guidance  and  directives  will  correct  and  standardize  how  the  planning  portion  of  risk 
management  is  executed  at  the  project  levels. 
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After  correcting  Air  Force  documentation,  the  next  challenge  will  be  to  actually 
achieve  effective  and  consistent  risk  management.  Baram  (2003)  proposes  a  method  of 
project  management  oversight  as  an 

established  practice  where  a  review  by  an  independent  (non-project  personnel)  is 
prepared  to  examine  the  “risk  status”  of  the  project  (time,  cost,  and  profit)  at 
selected  points  in  time,  and  possibly  explore  potential  hidden  problems  that 
should  be  addressed  and  proactively  reported  to  the  top  management  (p.  1). 

This  approach  directly  addresses  the  dilemma  identified  by  Kasperson  and  Kasperson 

(1996)  in  which  program  managers  and  engineers  are  reluctant  to  report  unfavorable  risk 

to  upper  management  and  customers  for  fear  of  deleterious  consequences.  Baram  (2003) 

describes  motives  as  to  why  program  managers  and  engineers  are  reluctant  to  provide  bad 

RM  reports,  as  well  as  issues  of  reactive  versus  proactive  reporting,  lack  of  experience, 

non-objective  reporting,  and  internal  politics  and  purports  the  project  management 

oversight  method  as  the  panacea.  Yet  several  immediate  concerns  with  project 

management  oversight  come  to  mind.  For  instance,  an  organization  would  have  to  be 

willing  and  able  to  dedicate  a  person  to  a  project  oversight  position.  Perhaps 

implementing  such  a  position  would  seem  distrustful  and  work  against  team  or 

organizational  cohesion. 

In  consideration  of  another  RM  process  solution,  Greiner  et  al.  (2002)  identified  a 
lack  of  effective  tools  available  for  the  execution  of  risk  management  through  a  series  of 
structured  interviews  of  key  members  of  the  multi-functional  Air  Force  Corporate 
Structure  (AFCS).  The  AFCS  is  the  key  unit  charged  with  making  decisions  regarding 
resource  allocation  against  all  Air  Force  activities,  including  weapon  system  acquisition 
programs.  The  tools  sought  after  to  fill  the  void  and  improve  Air  Force  Risk 
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Management  were  defined  as  “those  products  or  processes  that  provide  the  decision 
maker  with  a  structured  and  interactive  approach  to  accessing  and  interpreting  data,  with 
a  primary  goal  of  increasing  the  objectivity  within  the  decision  making  process”  (Greiner 
et  al.,  2002,  p.  129).  The  literature  evidences  a  disconnect  caused  by  a  PRM  tool  void  but 
Air  Force  Instruction  63-101/20-101,  designates  Active  Risk  Manager  (ARM)  as  the 
“current  standard  tool  to  manage  program  risks”  (2013,  p.  25).  Perhaps  the  issue  is  not  a 
lack  of  tools,  rather,  ARM  could  be  inadequate  per  the  definition  provided  by  Greiner  et 
al.  (2002). 

Langbein  (2005)  evaluates  ARM  as  focused  on  qualitative  risk  analysis  as  an 
expensive,  standalone  system  with  limited  or  awkward  interfaces  to  other  project 
management  software  applications.  It  is  the  author’s  experience  that  ARM  is  also  too 
expensive  to  distribute  to  program  managers  and/or  systems  engineers  who  are  charged 
with  the  responsibility  of  risk  management  and,  due  to  its  complexity,  necessitates  a 
dedicated  ARM  expert.  Though  there  are  not  any  additional  direct  assessments  of 
ARM’s  effectiveness  within  the  literature,  improvement  areas  can  be  logically  deduced. 
For  instance,  ARM  is  further  limited  because  it  cannot  interface  with  simple  tools 
(Langbein,  2005).  The  Author  can  confirm  that  Microsoft  Office  tools  are  prolific  and 
consistently  used  to  include  presentations  to  stakeholders  using  Microsoft  PowerPoint.  In 
addition,  modeling  is  implemented  using  Visio  and/or  a  Department  of  Defense 
architecture  framework  (DoDAF)  program.  The  takeaways  are  that  ARM  has  better 
potential  to  be  used  if  it  is  simple  like  the  tools  currently  used  (Tiwana  &  Keil,  2004)  and 
can  interface  with  these  tools,  as  the  objective  would  not  involve  replacing  them.  To  be 
effective,  ARM  would  need  to  be  applicable  to  the  project  and  enterprise  levels  of  RM, 
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and  its  internal  functions  would  restrict  the  user  to  compliance  with  USAF  and  DoD 
publications  even  while  remaining  flexible  enough  for  applicability  across  various  types 
of  projects.  However,  the  author  is  in  agreement  with  the  Husby,  Brede,  and  Tendal 
(1996)  article  on  the  perspective  that  “software  tools  are  very  useful  and  time  saving  but 
success  comes  from  the  project’s  team’s  use  of  these  tools  and  a  commitment  to  the  risk 
management  process  and  not  just  from  the  tools”  (p.  15-17). 

In  terms  of  tools,  another  deficiency  can  be  deduced  from  the  literature  regarding 
the  Air  Force’s  creation  of  DoDAF  views.  Despite  the  author’s  use  of  various  DoDAF 
programs  in  the  Air  Force,  there  has  yet  to  exist  a  DoDAF  framework  that  accounts  for 
risk  attributes  in  design  and  modeling.  Once  management  begins  treating  risk  as  a 
system  attribute  the  process  of  risk  management  will  enter  a  new  era  (Dagli,  2007).  In 
regards  to  the  latest  DoDAF  version  2.02,  the  DoD  Deputy  Chief  Information  Officer 
describes  that  DoDAF  conformance  is  achieved  when,  “the  data  in  a  described 
architecture  is  defined  according  to  the  DM2  concepts,  associations,  and  attributes”  (p. 

1). 

Summary 

Review  of  the  literature  provides  an  understanding  of  the  need  and  intent  behind 
Air  Force  PRM.  Trends  were  identified  which  define  a  general  approach  to  Air  Force 
PRM  but  diverge  where  consensus  is  critical.  Ambiguity  combined  within  DoD  and  Air 
Force  publications  seems  to  be  a  foundational  point  of  failure  in  regards  to  the  successful 
execution  of  PRM.  From  a  macro  level,  consistent  data  is  key  to  forming  enterprise 
strategy.  Additional  contributors  to  the  failure  Air  Force  PRM  were  considered  based  on 
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evidence  found  in  the  literature  and  the  author’s  observations  in  the  field  of  study.  The 
case  studies  presented  provide  further  support  to  some  of  the  contributors  but  there  is  still 
a  lack  of  data  to  confidently  determine  the  underlying  phenomena.  Effective  risk 
management  is  essential  to  the  continued  success  of  complex  systems;  contrarily,  if  not 
executed  properly,  it  can  be  the  false  safety  blanket  that  leads  to  catastrophe. 
Subsequently,  the  field  of  study  must  be  queried  in  pursuit  of  complete  data. 
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III.  Methodology 


Chapter  Overview 

Based  on  the  author’s  experiences  and  the  literature  reviewed,  a  working 
explanation  can  be  formed  to  articulate  inconsistent  PRM  across  the  Air  Force. 

However,  there  are  critical  aspects  to  the  explanation  that  require  further  support. 
Through  inductive  reasoning,  it  can  be  assumed  that  the  phenomenon  occurring  in 
comparable  settings  also  applies  not  only  to  AF  PRM  groups  and  their  projects,  but  also 
to  a  lack  of  specificity  concerning  the  Air  Force  environment.  For  example,  a  portion  of 
the  projects  referenced  in  the  literature,  though  strongly  comparable,  were  not  actual  Air 
Force  projects. 

To  gamer  the  support  and  instill  the  confidence  necessary  to  drive  corrective 
action,  research  will  be  employed  directly  in  the  field  of  study  to  further  investigate 
inconsistent  Air  Force  PRM.  Due  to  the  multitude  of  variables  and  subjective  nature  of 
the  study,  a  qualitative  research  approach  will  be  used  to  gather  data.  The  grounded 
theory  method  is  a  type  of  qualitative  research  that  will  anchor  the  previous  explanations 
in  data  gathered  from  interviews  conducted  directly  in  the  field  of  study.  The  interview 
protocol  (see  Appendix  A)  will  be  designed  to  impartially  allow  the  underlying 
phenomenon  behind  inconsistent  PRM  to  emerge  through  the  subject’s  answers.  In  this 
effort,  a  technique  referred  to  as  coding  will  be  used  to  process  the  data  for  patterns  and 
themes  that  evidence  the  underlying  phenomenon.  Overall,  the  research  strategy 
accounts  for  the  complex  nature  of  Air  Force  PRM  and  anticipates  an  equally  complex 
set  of  data  from  which  to  derive  an  explanation  for  the  inconsistencies  observed. 
Overarching  Methodological  Approach 
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The  evidence  that  reveals  why  risk  management  is  implemented  a  certain  way, 
and  the  resultant  decisions  and  actions  thereafter,  trace  back  to  the  complex  cognitive 
processes  of  risk  managers  and  those  who  influence  the  process.  It  is  then  necessary  to 
understand  the  relationship  between  tools  and  processes  from  the  users’  perspective  to 
assess  effectiveness  and  deficiency.  That  said,  effectiveness  of  tools  and  processes  can 
be  measured  by  what  degree  they  effectively  meet  the  intended  purpose.  For  instance,  a 
process  on  paper  can  appear  thorough  and  objective  but  is  useless  if  the  intended  users 
choose  not  to  use  it.  A  qualitative  study  to  explore  how  risk  managers  employ  processes 
and  tools  will  reveal  what  currently  works  and  what  does  not.  The  patterns  and  trends 
identified  will  help  direct  further  iterations  of  tools  and  processes  towards  effects-based 
risk  management.  To  better  understand  the  patterns  and  trends  within  risk  management, 
a  qualitative  grounded  theory  methodological  approach  will  be  used  to  gather  and  analyze 
data. 

John  Creswell  notes  that  “ qualitative  research  begins  with  assumptions,  a 
worldview,  the  possible  use  of  a  theoretical  lens,  and  the  study  of  research  problems 
inquiring  into  the  meaning  individuals  or  groups  ascribe  to  a  social  or  human  problem” 
(2007,  p.  37).  Following  suit  with  Creswell’s  description  and  through  the  supporting 
literature,  a  “problem”  has  been  defined,  “assumptions”  have  been  made  about  the  cause, 
and  it  is  necessary  to  validate  those  assumptions  by  “inquiring”  those  individuals  who  are 
involved  with  the  RM  tools  and  processes.  Complexity  is  another  factor  that  drives  the 
decision  for  a  qualitative  research  approach  (Creswell,  2007).  Even  though  the 
assumptions  thus  far  are  logically  grounded  in  the  literature  and  the  author’s  experience, 
they  are  compounded,  based  on  a  multitude  of  interrelated  variables  that  create  the 
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deficiencies  observed.  Consequently,  the  understanding  of  the  issues  at  hand  will  most 
likely  be  complex  and  full  of  detail  (Creswell,  2007).  Creswell  explains  that  “this  detail 
can  only  be  established  by  talking  directly  with  people,  going  to  their  homes  or  places  of 
work,  and  allowing  them  to  tell  the  stories  unencumbered  by  what  we  expect  to  find  or 
what  we  have  read  in  the  literature”  (2007,  p.  40).  Obtaining  data  from  several  sources, 
including  individuals  and  publications,  is  an  established  qualitative  data  gathering 
technique  (Creswell  &  Clark,  201 1).  Evidence  gathered  from  different  sources  have  been 
analyzed  for  codes  or  themes.  It  is  important  to  realize  and  avoid  the  potential  for  the 
literature  reviewed  and  the  author’s  experiences  to  influence  the  data  gathering  and 
analysis  process. 

In  qualitative  research  validity  is  more  of  the  primary  concern  than  reliability,  as 
validity  is  used  to  determine  whether  information  provided  by  research  subjects  is 
accurate  and  credible  (Lincoln  &  Guba,  1985).  The  challenge  with  qualitative  inquiry  of 
the  cognitive  processes  from  the  users’  perspective  is  drawing  out  and  capturing  true 
accounts  of  their  motives  and  understandings  within  the  risk  management  process.  In 
other  words,  the  users’  cognitive  phenomenon  must  be  drawn  out  and  espoused  in  a 
manner  to  promote  as  much  accuracy  and  trueness  as  possible.  In  this  effort, 
consideration  must  be  given  to  the  possibility  of  intentional  and  unintentional  deviations 
from  reality.  As  evidenced  by  the  literature,  bias  is  a  factor  to  be  considered  when 
analyzing  risk  management  and  information  gathered  from  the  community  has  the 
potential  to  carry  the  same  characteristics.  In  addition,  considering  that  the  very  nature  of 
risk  management  is  based  on  assumptions,  accounts  from  the  risk  management 
community  have  the  potential  to  contain  unintentional  erroneous  data.  As  explained  later 
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in  this  chapter,  controls  must  therefore  be  put  into  place  to  identify  and  omit 
noncontributing  data. 

Specific  Methodological  Technique 

Qualitative  research  can  cater  to  a  diverse  range  of  studies  of  varying  form, 
technicality,  complexity,  and  social  discipline  (Creswell,  2007).  Among  the  many 
techniques  described  in  the  literature  to  tailor  qualitative  research  to  a  specific  study,  five 
were  considered:  Narrative  Analysis,  Phenomenology,  Ethnography,  the  Case  Study,  and 
Grounded  Theory  (Creswell,  2007).  Comparatively,  the  approaches  considered  all  begin 
with  a  research  problem  from  which  research  questions  are  formed  in  an  effort  to  explain 
an  underlying  phenomenon.  Data  is  then  gathered  and  analyzed  for  evidence  that  may 
answer  the  research  question(s).  Specific  to  this  thesis,  the  problem  identified  is  a  lack  of 
consistency  in  Air  Force  risk  management,  and  the  research  question  seeks  to  identify  the 
contributors  to  that  phenomenon.  The  data  gathering  process  was  initiated  by  analyzing 
the  literature  germane  to  the  stated  problem  and  keyed  in  on  a  set  of  potential 
contributors  to  Air  Force  RM  inconsistency  outlined  in  Chapter  2.  The  subsequent 
qualitative  research  technique  will  extract  information  from  the  domains  where  the 
contributing  phenomenon  exists  to  either  bolster  or  disprove  what  was  evidenced  by  the 
literature.  The  five  qualitative  techniques  considered  endeavor  to  collect  data  from  the 
domain  of  study;  that  said,  the  use  of  interviews,  observations,  documents,  and 
audiovisual  materials  are  the  primary  forms  of  data  collection  with  respect  to  the 
aforementioned  five  techniques  (Creswell,  2007).  However,  the  five  fundamentally 
differ  in  their  foci  and  is  thus  the  basis  for  the  author’s  decision  on  which  technique  to 
implement  for  the  research.  That  said,  the  author  concluded  that  the  grounded  theory 
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technique  is  best  suited  to  determine  the  contributors  to  the  inconsistency  in  Air  Force 
RM. 

The  objective  of  using  the  grounded  theory  technique  is  to  verify  the  assumptions 
made  from  the  literature  discussed  in  Chapter  2,  especially  with  respect  to  information 
taken  directly  from  the  Air  Force  domain  of  risk  management.  The  selection  criteria  for 
choosing  the  grounded  theory  technique  accounted  for  the  study’s  focus,  type  of  problem 
best  suited  for  design,  discipline  background,  and  unit  of  analysis  (Creswell,  2007).  In 
line  with  Creswell’s  criteria  and  specific  to  the  grounded  theory  technique,  the  “focus” 
will  be  “developing  a  theory  grounded  in  data  from  the  field”  (Creswell,  2007,  p.  78)  of 
Air  Force  project  risk  management.  The  “type  of  problem  best  suited  for  design  [is] 
grounding  the  theory  in  the  views  of  participants”  (Creswell,  2007,  p.  78)  in  the  form  of 
interviews  taken  from  Air  Force  project  risk  management  key  personnel.  The  discipline 
background  will  draw  from  sociology  as  it  relates  to  project  risk  management  teams  and 
the  processes  and  tools  they  are  bound  to  (Creswell,  2007).  Lastly,  it  is  the  case  that  the 
grounded  theory  technique  is  best  used  for  “studying  a  process,  action,  or  interaction 
involving  many  individuals”  (Creswell,  2007,  p.  78)  which,  in  the  current  instance, 
applies  to  the  study  of  interaction  between  many  Air  Force  project  risk  management 
individuals  and  the  risk  management  processes  and  tools. 

Participants 

One  of  the  suggested  methods  to  gather  data  from  the  field  involves  inquiry  in  the 
form  of  interviews  from  those  key  persons  or  groups  germane  to  the  study  and  theory 
being  investigated.  Creswell  (2007)  specifies  that  data  from  the  field  should  be 
especially  grounded  in  the,  “actions,  interactions,  and  social  processes  of  people”  (p.  63). 
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Specific  to  the  grounded  theory  technique,  the  target  audience  should  be  those  that  have 
experienced  the  process  and  are  therefore  affected  by  and/or  affect  the  underlying 
phenomenon  (Creswell  2007). 

The  Risk  Management  Guide  for  DOD  Acquisition  states  that  “since  risk  can  be 
associated  with  all  aspects  of  a  program,  it  is  important  to  recognize  that  risk 
identification  is  part  of  the  job  of  everyone  and  not  just  the  program  manager  or  systems 
engineer”  (DOD  2006,  p.  1).  However,  in  the  author’s  experience,  Air  Force  program 
managers  are  the  most  involved  with  leading  project  risk  management  and  are 
furthermore  required  to  do  so.  For  this  reason,  the  author  identified  Air  Force  program 
managers  for  interviews.  In  order  to  mitigate  potential  bias,  it  is  necessary  to  investigate 
if  indeed  Air  Force  program  managers  implement  risk  management  as  observed  and 
required.  It  is  also  possible,  especially  in  larger  programs  and  projects,  that  program 
managers  are  accompanied  by  a  systems  engineer  and/or  risk  management  subject  matter 
experts  (SME)  who  may  be  more  directly  involved  with  implementing  system 
engineering  techniques  and  tools.  Initially  targeting  program  managers  is,  in  essence,  a 
top-down  or  macro  level  approach  that  is  expected  to  lead  to  interviews  of  more 
technically  focused  risk  management  positions  when  they  exist.  Inquiry  of  systems 
engineers  and  risk  management  SMEs  would  offer  invaluable  insight  given  their  more 
direct  focus  on  risk  management  processes  and  tools. 

It  is  also  important  to  understand  the  relationship  between  the  systems  engineer 
and  program  manager  when  the  two  positions  exist  simultaneously.  The  Air  Force 
directs  its  program  managers  to  “establish,  use,  and  maintain  an  integrated  risk 
management  process”  (DOD  2006,  p.  26)  among  many  other  responsibilities  to  include 
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the  management  of  cost,  schedule  and  performance.  The  Defense  Acquisition  Guidebook 
addresses  both  roles  when  stating  that  “effectively  managing  risks  helps  the  Program 
Manager  and  Systems  Engineer  develop  and  maintain  a  system’s  technical  performance, 
and  ensure  realistic  life-cycle  cost  and  schedule  estimates”  (DAU  2013,  p.  140).  It  is  the 
author’s  experience  that  a  dedicated  program  manager  is  required,  and  that  systems 
engineers  are  assigned  as  necessary.  Otherwise,  in  the  absence  of  a  systems  engineer,  the 
program  manager  becomes  dual  hatted  and  oversees  systems  engineering  functions.  It 
would  be  insightful  to  track  the  relationship  between  the  dual-hatted-program-manager 
versus  a  dedicated-systems-engineer  model  in  terms  risk  management  performance.  It 
would  also  be  insightful  to  investigate  the  role  of  SMEs  when  possible.  It  is  the  author’s 
experience  that  dedicated  project  risk  management  SMEs  tend  to  be  contractors  who 
introduce  techniques  and  practices  originating  from  their  respective  companies.  In  this 
respect,  it  would  be  insightful  to  obtain  their  views  on  their  company’s  project  risk 
management  approach  versus  the  Air  Force’s  approach  and  subsequent  methods  for 
integration. 

It  could  be  argued  that  persons  holding  roles  and  positions  at  higher  levels  in  the 
risk  management  hierarchy  should  be  queried  for  their  perspective.  For  instance,  it  can 
be  deduced  that,  since  Air  Force  enterprise  level  risk  management  depends  on  project 
level  risk  management  input,  it  would  be  insightful  to  obtain  the  views  of  enterprise  risk 
managers.  However,  the  focus  is  on  the  phenomenon  that  contributes  to  inconsistent  risk 
management  assessment  at  the  project  level.  The  confirmation  therefore  is  a  predecessor 
to  an  investigation  on  the  affects  of  inconsistent  project  risk  management  on  enterprise 
risk  management.  Although  enterprise  risk  managers  most  likely  have  prior  knowledge 
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or  indirect  insight  of  the  project  level  of  risk  management,  the  most  applicable  and  direct 
insight  will  be  obtained  from  program  managers,  systems  engineers,  and  associated 
SMEs  at  the  Air  Force  project  level  of  risk  management. 

Because  the  success  of  Air  Force  projects  and  programs  are  dependent  on 
effective  communication,  it  will  be  feasible  to  identify  and  contact  program  managers, 
systems  engineers,  and  associated  SMEs  for  interviews.  It  is  necessary  to  be  able  to 
identify  positions  within  programs  in  order  to  cross-communicate  between  projects  and 
integrated  project  teams  (IPT).  For  this  reason,  it  is  the  author’s  experience  that  positions 
and  associated  contact  information  is  often  made  available.  However,  it  is  also  the 
author’s  observation  that,  due  to  security  concerns,  position  and  contact  information  is 
often  restricted  to  those  authorized  or  with  access  to  the  restricted  networks  holding  the 
information.  Furthermore,  the  means  to  make  contact  and  communicate  with  those  who 
hold  positions  for  or  within  the  military  is  restricted  in  the  same  way.  Information 
restrictions  in  this  regard  will  not  be  a  hindrance  because  the  author,  who  will  be  the  only 
person  conducting  interviews,  is  an  active  duty  member  of  the  Air  Force  who  has 
authorized  access  to  the  networks  holding  the  necessary  contact  information. 

Two  methods  will  be  used  to  identify  and  contact  program  managers,  systems 
engineers,  and  associated  SMEs  for  interviews.  The  first  method  will  utilize  the  Air 
Force  Portal  to  find  DOD  websites  with  organizational  charts  and  contact  information. 
Phone  calls  will  be  the  preferred  method  of  contact.  When  phone  numbers  are  not 
available,  the  Air  Force  global  contact  list  found  on  Microsoft  Outlook,  which  is  a 
standard  and  prolific  DOD  emailing  system,  will  be  used  to  look  up  both  email  and  phone 
number(s)  associated  with  names.  The  second  method  will  use  the  direct  references  of 
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those  the  author  has  and  currently  works  with. 

Data  Collection  Process 

The  interviews  by  which  subjects  are  queried  are  considered  by  many  sources 
both  an  art  and  a  science  to  promote  an  account  that  is  accurate,  can  be  trusted,  and  is 
credible  (Creswell,  2007;  Creswell  &  Plano  Clark,  2011;  Lincoln  &  Guba,  1985).  For 
instance,  in  the  literature,  interviews  have  been  described  as  an  encounter  (Goffirum, 
1967),  a  social  performance  (Goffman,  1959),  a  face-to-face  interactionary  performance 
(Babbie,  1992,  1998),  and  creative  interaction  (Douglas,  1985).  In  this  respect, 
consideration  is  given  to  more  than  just  the  set  of  questions  and  answers  to  achieve 
notable  data.  Field  research,  such  as  conducting  interviews,  is  often  divided  into  two 
phases:  getting  in  and  analysis  (Shaffir  et  al.,  1980).  That  said,  one  of  the  first 
considerations  upon  initial  contact  of  the  targeted  subjects,  other  than  coordination  of  a 
start  time,  is  the  medium  through  which  communication  will  take  place.  An  interview 
conducted  in  person  is  preferred  because  face-to-face  interaction  allows  nonverbal  cues 
to  be  perceived  by  the  interviewer  and  used  to  pace  the  query  and  influence  the  proper 
direction  of  discussion  (Berg  &  Lune  2012).  However,  because  the  targeted  sample  will 
represent  different  units  and  missions  across  the  Air  Force,  the  majority  of  the  subjects 
will  not  be  locally  available  to  support  in-person  interviews.  Air  Force  personnel  are 
geographically  dispersed  around  the  world  in  order  to  support  global  omnipresent 
military  operations.  As  a  primary  example,  the  author,  who  will  be  the  only  person 
conducting  interviews,  must  overcome  the  restrictions  imposed  by  an  Air  Force 
deployment.  Considering  that  a  large  enough  pool  of  subjects  may  not  be  available  on  a 
small  forward  deployed  operating  base,  the  author  will  need  to  reach  out  via  phone  to 
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conduct  a  portion  of  the  subject’s  interviews. 

In  addition  to  giving  consideration  to  the  logistics  surrounding  the  interview,  a 
determination  must  be  made  about  which  interviewing  technique  to  employ.  Three 
techniques  were  considered:  the  standardized  (i.e.,  formal  or  structured)  interview,  the 
unstandardized  (i.e.,  informal  or  non-directive)  interview,  and  the  semi-standardized  (i.e., 
guided  semi- structured  or  focused)  interview  (Babbie,  1995;  Denzin,  1978;  Frankfort- 
Nachmias  &  Nachmias,  1996;  Gorden,  1987;  Nieswiadomy  1993).  The  semi- 
standardized  interview  technique  was  chosen  as  the  best  approach  considering  the  subject 
matter  and  audience,  as  it  lies  between  the  extremes  of  completely  standardized  and 
completely  unstandardized  interviewing  (Berg  &  Lune  2012).  On  one  hand,  the 
questions  are  queried  in  a  systematic  and  consistent  order,  but  on  the  other  hand,  the 
interviewer  is  allowed  the  freedom  to  digress  and  probe  far  beyond  the  answers  to  the 
prepared  and  standardized  questions  (Berg  &  Lune  2012).  Conducting  interviews  in 
person  and  using  the  semi- standardized  technique  gives  the  interviewer  the  flexibility  to 
capitalize  on  opportunities  to  probe  for  a  better  understanding  of  the  underlying 
phenomenon. 

Because  there  is  flexibility  built  into  the  semi- standardized  interviewer  approach, 
open-ended  questions  can  be  used  to  solicit  descriptive  answers  that  will  set  the  stage  for 
further  question  formulation.  As  suggested  by  Selltiz  et  al.  (1959),  Spradley  (1979), 
Patton  (1980),  and  Polit  and  Hungler  (1993),  the  development  of  a  set  of  interview 
questions,  also  termed  a  schedule,  will  begin  with  an  outline  that  lists  all  the  broad 
categories  relevant  to  the  study.  Four  broad  categories  will  be  used  to  form  the  initial 
outline:  source  of  risk  management  requirement  and  process;  emphasis  on  having  a  risk 
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management  process  and  source  of  that  emphasis;  actual  contribution  of  the  RM  process 
in  handling  risks  and  the  subsequent  positive  effects,  and;  the  causes  to  risk  management 
ineffectiveness.  The  preliminary  listing  of  categories  helps  to  conceptualize  the  general 
format  of  the  schedule.  The  schedule  is  located  in  Appendix  A  of  this  thesis. 

After  an  outline  and  broad  categories  of  questions  have  been  established,  sets  of 
specific  questions  relevant  to  each  category  will  be  developed.  The  type  of  questions 
developed  plays  an  important  role  in  the  interviewer’s  strategy  to  obtain  full  and  accurate 
data  pertaining  to  the  study.  The  four  types  of  questions  identified  in  the  literature  were 
essential  questions,  extra  questions,  throwaway  questions,  and  probing  questions  (Berg  & 
Lune  2012).  In  accordance  with  the  purpose  described  in  the  literature,  throwaway 
questions  will  be  used  at  the  beginning  of  the  interview  to  develop  rapport,  set  the  pace, 
allow  for  change  in  focus,  draw  out  a  complete  story  from  research  subjects,  and  cool  out 
the  subject  if  necessary  (Becker,  1963;  Goffman,  1967).  Because  the  Air  Force  is  a 
close-knit  community  and  the  interviewer  is  an  Air  Force  member  and  the  subject  will  at 
least  be  affiliated  with  the  Air  Force,  it  will  be  natural  to  discuss  job  title,  role,  mission, 
and  the  like.  Job  related  questions  meet  the  objectives  stated  in  the  literature  for 
throwaway  questions  and  establishes  common  ground  between  the  interviewer  and 
subject.  The  trust  and  report  that  ensues  will  lead  to  more  complete  and  accurate 
answers.  Additionally,  job  related  information,  when  compared  to  subsequent 
information  in  the  interview,  is  likely  to  reveal  an  insightful  pattern  and  may  not  have  to 
be  thrown  away  after  all.  After  the  subject’s  general  occupational  background  and 
experience  has  been  discussed  and  rapport  and  common  ground  has  been  establish, 
essential  questions  will  be  asked. 
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While  throwaway  questions  will  only  exist  in  the  initial  portion  of  the  survey, 
each  category  thereafter  will  begin  with  essential  questions  intermixed  with  probing 
questions  followed  by  extra  questions.  “Essential  questions  exclusively  concern  the 
central  focus  of  the  study”  (Berg  &  Lune  2012,  p.  75)  and  will  serve  to  elicit  specific  data 
that  I  assume  will  bring  light  to  the  underlying  phenomenon.  A  series  of  general  probing 
questions  will  be  prepared  and  ready  to  implement  the  moment  the  need  arises  to  further 
explore  a  subject’s  statement.  The  extra  follow-up  questions  are  important  to  verify  the 
reliability  of  the  essential  questions.  For  this  reason,  there  will  most  likely  be  similarities 
between  the  essential  questions  and  extra  questions.  As  mentioned  previously,  in 
qualitative  research,  the  key  is  on  validating  the  data  rather  than  ensuring  reliability. 

By  design,  there  is  ample  opportunity  to  sufficiently  retrieve  data  from  the 
subjects  interviewed.  Each  category  will  have  a  set  of  questions  that  pinpoints  the 
phenomenon  that  contributes  to  inaccurate  risk  assessment.  The  semi-structured 
interview  approach,  when  coupled  with  probing  questions,  will  allow  the  interviewer  to 
go  beyond  the  initial  set  of  questions  to  pursue  key  data  until  the  four  categories  are 
sufficiently  covered.  Though  somewhat  redundant,  the  extra  questions  will  serve  as  a 
first  step  in  data  validation  by  cross-referencing  previous  questions  to  indicate 
consistency  in  answers  given  by  respondents,  or  the  lack  thereof. 

Data  Analysis  Process 

The  raw  data  gathered  from  interviews  will  be  analyzed  relative  to  the  literature 
and  the  author’s  observations.  A  methodical  approach  in  this  regard  will  set  the  stage  for 
grounding  the  author’s  explanations  in  the  qualitative  interview  data  through  analytic 
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induction.  Analytic  induction  will  reveal  patterns  in  the  data  that  will  lend  themselves  to 
the  creation  of  categories.  These  categories  must  be  grounded  in  the  data  from  which 
they  emerged  (Denzin,  1978;  Glaser  &  Strauss,  1967).  In  this  effort,  the  data  will  be 
understood  and  sorted  by  thematic  group  types  relevant  to  the  phenomenon  under 
investigation.  For  instance,  it  will  be  useful  to  distinguish  between  risk  management 
processes  derived  from  Air  Force  guidance  versus  those  imposed  by  contractors.  Doing 
so  may  evidence  a  lack  of  ineffectiveness  in  Air  Force  RM  guidance  or  inconsistencies 
between  RM  processes  by  different  contractors. 

Once  the  interview  data  was  sorted,  themes  relevant  to  the  scope  of  the 
investigation  were  developed  through  the  process  of  inductive  coding  techniques.  The 
inductive  coding  techniques  used  were  first  articulated  in  1967  by  Glaser  and  Strauss  but 
best  articulated  in  1990  by  Strauss  and  Corbin.  Specifically,  Anselm  Strauss,  his 
colleagues,  and  other  methodologists  outline  a  three-step  process  to  accomplish 
qualitative  in-depth  interview  data  analysis  through  inductive  coding.  Berg  describes  the 
process  of  coding  as  a  funnel  where  “you  begin  with  a  wide  opening,  a  broad  statement; 
narrow  the  statement  throughout  the  body  by  offering  substantial  backing;  and  finally,  at 
the  small  end  of  the  funnel  present  a  refined,  tightly  stated  conclusion”  (Berg  2001,  252). 
The  three  steps,  from  broadest  to  most  refined,  include  open  coding,  axial  coding,  and 
selective  coding.  Throughout  the  process,  the  primary  research  question  can  be  a  framing 
mechanism  from  which  the  themes,  or  the  answers  to  the  research  questions,  can  emerge. 

Open  coding.  As  the  broadest  and  first  step,  open  coding  will  serve  as  a  ‘first 
pass’  or  a  ‘first  read’  of  the  interview  notes  to  get  a  feel  for  the  data.  In  essence,  open 
coding  will  initiate  immersion  in  the  data  to  gain  a  deep  and  thorough  understanding, 
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which  is  the  primary  objective  of  the  coding  process.  As  Neuman  (2000)  describes,  open 
coding  “brings  themes  to  the  surface  from  deep  inside  the  data”  (p.  422).  To  conduct 
open  coding,  four  guidelines  prescribed  by  Strauss  (1987,  p.  30)  will  be  followed.  First, 
the  data  will  be  used  to  form  specific  and  consistent  questions  toward  meeting  the 
original  objective  of  the  research  question  (Berg,  2001).  Specific  to  this  study,  questions 
will  be  aimed  at  identifying  the  phenomenon  that  contributes  to  RM  inaccuracy  or  the 
lack  thereof.  Second,  the  data  will  be  analyzed  minutely  to  identify  all  relevant 
categories,  incidents,  and  interactions  to  ensure  extensive  theoretical  coverage  and 
grounding  until  a  set  of  codes  become  repetitious  (Berg,  2001).  Third,  the  coding  of  the 
data  will  be  interrupted  frequently  to  write  notes  on  ideas,  notions  or  thoughts  (Berg, 
2001).  Doing  so  will  ensure  that  ideas  triggered  by  the  data,  though  not  instantaneously 
key,  will  be  captured  to  complete  the  explanation  later.  Fourth,  the  relevance  of  variables 
will  not  be  assumed  unless  the  data  supports  it  (Berg,  2001).  Preventing  precognitions 
from  influencing  the  assessment  of  the  data  ensures  that  the  final  findings  are  as  accurate 
as  possible.  Once  key  words  and  phrases  have  been  extracted  from  the  data  through  open 
coding,  the  assessment  will  proceed  to  the  second  step:  axial  coding,  or  developing 
coding  frames. 

Axial  coding  /  Coding  frames.  Neuman  (2000)  describes  axial  coding  as  a 
‘second  pass’  that  allows  the  researcher  to  make  connections  between  themes  that 
emerged  in  the  open  coding  process.  Neuman  further  clarifies  that  axial  coding  provides 
the  data  analyst  the  opportunity  to  focus  on  the  initially  coded  themes  more  than  on  the 
primary  data  itself.  More  specifically,  Strauss  and  Corbin  (1990)  describe  axial  coding  as 
forming  connections  between  the  key  words,  phrases,  and  themes  developed  via  open 
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coding.  Berg  and  Lune  (2012)  offer  a  similar  viewpoint  but  refer  to  axial  coding  as 


developing  coding  frames .  Although  essentially  the  same  as  axial  coding,  the  use  of 
coding  frames  allows  for  a  more  broadly  based  connection  between  the  themes  developed 
between  open  coding.  In  other  words,  axial  coding  tends  to  have  an  unlimited  number  of 
potential  connections  between  themes;  in  contrast,  coding  frames  tend  to  limit  the 
number  of  connections  developed  among  themes  discovered  in  open  coding.  Following 
suit  with  the  techniques  prescribed  in  the  literature,  coded  observations  will  be  cross- 
referenced  for  notable  similarities  or  discrepancies  and  then  compared  across  themes  to 
see  how  each  is  either  similar  or  different. 

Selective  coding.  Selective  coding  is  the  third  aspect  of  qualitative  inductive  data 
analysis.  It  will  serve  as  a  ‘third  pass’  through  the  data  to  see  if  the  previous  codes 
developed  in  open  and  axial  coding  are  sound  from  a  theoretical  standpoint.  The 
patterns,  themes  and  codes  developed  from  the  data  will  be  examined  in  the  context  of 
the  literature  and  theories  described  thus  far  to  see  if  there  are  connections  to  the  data 
(Berg  &  Lune  2012).  Neuman  (2000)  argues  that  selective  coding  can  be  used  to  see  if 
the  major  themes  and/or  concepts  are  either  (a)  relevant  within  the  context  of  the 
research,  or  (b)  can  be  used  to  build  up  an  explanatory  framework  or  theory. 


Ethical  Concerns 

There  was  little  to  no  risk  that  the  interview  designed  for  this  study  induced  an 
ethical  breach.  Fundamentally,  the  data  that  was  sought  after  in  the  schedule  was  purely 
related  to  organizational  practices  and  did  not  obtain  any  personal  identifiers  or 
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demographic  information.  Because  the  schedule  solicited  information  that  reflected  the 
organization  and  subject’s  attitude  towards  risk  management,  all  identifiers  were  kept 
confidential  and  stored  securely.  Specifically,  the  identifiers  were  only  known  and 
accessible  to  the  author  as  the  sole  researcher,  and  the  data  collected  was  kept  secure  and 
managed  according  to  the  recommendations  that  the  Institution  Review  Board  (IRB) 
made  concerning  the  interview  protocol.  If  a  subject’s  response  reasonably  placed  them 
at  risk  of  criminal  or  civil  liability  or  is  damaging  to  their  financial  or  personal  standing, 
employability,  or  reputation,  the  requirement  is  understood  that  an  adverse  event  report 
will  be  immediately  filed  with  the  institution’s  IRB  office.  At  the  conclusion  of  the 
study,  the  Principal  Investigator  (PI)  turned  over  all  de-identified  data  to  the  advisor,  and 
extraneous  copies  were  destroyed.  Stated  concisely,  the  interviews  conducted  for  this 
study  and  all  corresponding  efforts  were  done  in  accordance  with  the  institution’s  IRB 
regulations. 

Summary 

The  challenge  faced  in  this  particular  study  is  finding  the  key  details  necessary  to 
accept  or  reject  the  explanation  deduced  from  the  literature.  The  data  gathered  from 
interviews  must  be  tediously  scrutinized  and  sorted  to  reveal  the  phenomenon,  or  in  the 
case  of  this  study,  the  contributors  to  inconsistent  Air  Force  project  risk  management. 
Towards  this  end,  steps  will  be  taken  to  ensure  that  the  data  pool  is  as  robust  and  valid  as 
possible.  The  criteria  used  for  selecting  interview  subjects  will  target  those  who  are  in  a 
position  to  bare  the  underlying  truths  behind  risk  management  processes  in  the  Air  Force. 
The  potential  for  bias  will  be  mitigated  by  masking  the  subjects’  identities  and  associated 
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organizations  and  doubly  serves  to  comply  with  ethical  IRB  regulations.  Such 
considerations  of  extraneous  variables  enable  the  success  of  the  interview  process  and 
data  analysis. 

The  effectiveness  of  the  study  as  a  whole  will  hinge  on  the  success  of  data 
gathering  and  analysis  thereafter.  The  schedule  of  questions  will  be  created  as  a  way  to 
hone  in  on  the  explanations  derived  from  the  literature  earlier,  but  as  is  intended  with 
semi- structured  interviews,  will  allow  the  flexibility  to  discover  other  contributors  to 
inaccurate  risk  management.  While  the  interview  technique  will  allow  variation,  the  data 
analysis  process  will  be  methodic  and  structured  in  order  to  sort  the  complexity  expected 
within  the  data  collected.  The  overall  design  of  the  study  takes  into  account  the  unique 
characteristics  of  the  research  problem  and  is  grounded  in  proven  techniques. 
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IV.  Analysis  and  Results 


Chapter  Overview 

Twenty  subjects  who  fit  the  research  criteria  outlined  in  chapter  three  of  this 
thesis  were  interviewed  and  sufficient  amounts  of  raw  data  were  collected  to  answer  the 
investigative  questions  and  draw  conclusions.  A  description  of  the  subjects  interviewed 
will  be  provided  to  give  context  to  the  data  collected  and  analysis  of  that  data. 
Specifically,  the  description  will  explore  the  skill  sets,  mission  involvement,  experience, 
and  employment  conditions  unique  to  the  subjects  interviewed.  Descriptions  of  the 
subjects  interviewed  will  also  align  to  the  scope  of  the  research  study.  This  chapter  will 
provide  a  description  of  how  the  study  was  conducted,  what  data  was  gathered  and  how 
the  data  was  interpreted.  Finally,  all  components  surrounding  Air  Force  PRM  will  be 
available  for  discussions  and  conclusions. 

In  this  chapter,  it  will  become  evident  that  the  results  of  the  research  study  stem 
from  a  proven  research  approach,  i.e.,  grounded  theory,  which  has  been  tailored  and 
strategically  scoped  to  investigate  the  particular  gaps  of  knowledge  identified  regarding 
the  consistency  of  PRM  processes  in  the  Air  Force.  The  groundwork  will  then  be  set  to 
make  comparisons  between  the  data  gathered  from  interviews,  the  author’s  observations, 
and  the  findings  in  the  surrounding  literature.  A  comprehensive  review  of  the  research 
study  will  also  be  provided  to  clarify  if  risk  management  processes  are  truly  inconsistent, 
and  if  so,  whether  the  lack  of  normalization  impacts  decisions  made  at  the  enterprise 
level. 
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Research  Study  Scope 

The  research  study  conducted  endeavored  to  answer  the  following  five 
foundational  questions  posed  in  chapter  one  of  this  thesis: 

•  Is  project  risk  management  (PRM)  inconsistently  practiced  in  the  U.S.  Air  Force? 

•  What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

•  How  do  inconsistencies  in  Air  Force  PRM  processes  lead  to  inaccurate  risk 
assessments?  How  do  inconsistent  PRM  assessments  impact  executive  RM? 

•  What  contributes  to  Air  Force  PRM  inconsistencies? 

•  What  steps  can  be  taken  to  address  contributors  to  inconsistent  PRM  across  the 
Air  Force? 

The  interview  questions  were  derived  from  these  five  foundational  questions  in  a 
format  more  suitable  for  obtaining  complete  data  through  interview  discussions. 
Therefore,  the  five  foundational  questions  and  the  interview  set  both  address  the 
assumption  that  inconsistent  risk  management  practices  between  organizations  within  the 
Air  Force  can  be  attributed  to  a  combination  of  ambiguous  direction,  cognitive  biases, 
flawed  practices,  and  inadequate  tools.  The  results  of  the  research  study  will  be 
comparable  to  the  author’s  observations  and  the  literature  review  for  a  couple  of  reasons. 
First,  the  author’s  observations  made  directly  in  the  field  of  study  are  what  initiated  the 
project  and  led  to  the  five  foundational  questions.  Second,  the  research  study  shares  the 
same  objective  as  the  literature  review  in  its  attempt  to  answer  the  five  foundational 
questions.  Answering  the  five  foundational  questions  directly  addresses  the  assumptions 
made  about  inconsistent  risk  management  practices  in  the  Air  Force. 
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With  the  foundational  questions  finalized,  the  next  challenge  was  to  obtain 
credible  answers.  Obtaining  credible  answers  meant  interviewing  subjects  that  had  direct 
experience  with  the  risk  management  processes  in  the  Air  Force  field  of  acquisitions. 
Specifically,  the  criteria  for  choosing  interview  subjects  only  required  that  they  have  or 
had  worked  in  the  Air  Force  field  of  acquisitions  as  a  program  manager,  systems 
engineer,  and/or  contractor  PRM  SME.  There  was  no  preference  concerning  the  age  or 
sex  of  respondents  as  these  attributes  were  considered  to  be  non-relevant.  However, 
there  was  an  assumed  potential  for  patterns  to  emerge  between  the  interview  data  and  the 
subject’s  rank,  experience  level,  career  field,  mission  type/major  command  (MAJCOM), 
employer,  product,  and  product  budget.  The  recording  of  a  subject’s  attributes  germane 
to  the  study  was  done  during  the  initial  interview  session  as  singular  questions,  while 
other  attributes  were  best  investigated  as  part  of  the  in-depth  discussion. 

Method  Execution 

In  terms  of  the  method  planned,  the  research  study  achieved  all  the  objectives  by 
provoking  insightful  discussion  and  obtaining  data  that  revealed  the  underlying 
phenomenon  causing  inconsistent  PRM  processes  across  the  Air  Force.  The  threshold  set 
for  the  number  of  subjects  to  be  interviewed  ranged  from  fifteen  to  twenty,  and  the 
objective  to  interview  twenty  subjects  was  met.  While  the  criteria  set  for  subjects  was 
met  in  all  cases,  there  was  also  an  overarching  effort  to  have  a  broad  variance  of  certain 
subject  attributes.  The  result  of  achieving  this  goal  was  twofold:  1)  A  more 
comprehensive  representation  of  the  population  most  involved  with  Air  Force  PRM;  2) 
Insightful  patterns  were  found  in  the  association  between  the  subject  attributes  and  the 


56 


answers  provided.  The  study  targeted  Air  Force  acquisition  members  with  attributes 
ranging  in  seven  areas.  The  following  table  shows  the  seven  attributes  and  their 
respective  ranges  mapped  to  the  subjects  interviewed. 

Table  1 


Summary  of  Subject  Attributes 


ATTRIBUTE 

RANGE  and  COUNT 

Mission  Type 

Air  (4) 

Space (12) 

Cyberspace  (9) 

Employer 

Active  Duty  (9) 

Gov’t  Civilian  (9) 

Contractor  (5) 

Rank  (1) 

Fow  (6) 

Medium  (10) 

High  (4) 

Career  Field 

Systems  Engineer  (8) 

Program  Manager  (11) 

PRM  SME  (5) 

Product 

New  Tech  (9) 

Support  Equipment  (7) 

Weapon  System  (12) 

Product  budget 

<  $10M  (14) 

$10M  <  $100M  (9) 

>  $100M  (7) 

Experience  111 

High  (14)  Fow  (6) 

Key: 

(1)  Low:  1  Lt  and  below  or  GS/GG  1 1  and  below  or  contractor  equivalent;  Medium:  Capt 
through  Maj  or  GS/GG  12  through  13  or  contractor  equivalent;  High:  LtCol  through  Col 
or  GS/GG  14  through  15  or  contractor  equivalent. 

(2)  Low:  Experience  with  one  additional  unit  PRM  process  or  less;  High:  Experience 
with  two  additional  different  unit  PRM  processes  or  more 

Fortunately,  the  target  sample  was  readily  available  because  the  U.S.  Air  Force 

acquisitions  community  is  close-knit  and  has  a  global  index  of  all  members  and  their 

contact  information.  An  additional  technique,  that  proved  effective,  involved  asking  the 

subjects  to  recommend  other  members  who  fit  the  targeted  characteristics.  For  these 

reasons,  it  was  possible  to  interview  subjects  possesses  a  diverse  range  of  qualities  in  the 

seven  target  attribute  areas. 

Despite  the  advantages  mentioned,  the  need  to  reach  a  sample  of  subjects 
representative  of  Air  Acquisitions  as  a  whole,  and  therefore  different  mission  types, 
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meant  that  geographical  dislocation  prevented  in-person  communication  for  fourteen  out 
of  twenty  of  the  interviews.  At  first,  having  interviews  in-person  versus  over  the 
telephone  was  highly  preferred  for  establishing  report  and  building  trust.  However,  the 
camaraderie  and  commonality  between  Air  Force  members  countered  the  disadvantages 
inherent  in  over-the-phone  interviewing.  Such  elements  also  contributed  to  an  overall 
willingness  of  members  to  contribute  their  time  and  effort  towards  the  study.  Subjects 
seemed  to  be  further  motivated  to  partake  in  the  interview  once  made  aware  of  the 
discussion  topic.  The  subject  of  PRM  was  a  point  of  frustration  for  the  majority  of  the 
subjects,  especially  those  with  more  experience,  and  the  interview  sometimes  became  an 
opportunity  to  rant  and  vent  for  some  of  the  research  subjects. 

Despite  tendencies  for  the  topic  of  PRM  to  provoke  passionate  tangents  among 
some  of  the  research  subjects,  the  interview  set  served  to  solicit  the  intended  discussions 
and  information.  The  question-set  was  used  as  a  trial  four  times  with  colleagues  before 
actual  use  in  the  field,  and  as  a  result,  was  refined  to  serve  as  a  seamless  discussion 
prompt  to  ensure  all  questions  were  clearly  received  and  remained  unchanged  for  all 
interviews.  The  intention  was  to  conduct  all  interviews  over  the  phone  or  in  person  while 
capturing  the  raw  data  as  recordings.  However,  the  majority  of  the  subjects  had  fully 
booked  schedules,  and  six  out  of  the  twenty  subjects  requested  to  preview  the  question 
set  and  make  notes  prior  to  the  interview.  In  these  cases,  the  interview  became  more 
focused  on  the  questions  or  answers  that  were  not  fully  understood  or  that  were  deserving 
of  further  discussion.  Transcribing  the  interview  then  became  a  matter  of  expanding 
upon  the  subject’s  written  answers.  The  deviation  from  plans  in  this  manner  is  not 
suspected  to  have  reduced  the  validity  of  the  data  received. 
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From  an  ethical  standpoint,  it  was  important  to  ensure  the  subjects  anonymity  of 
concerning  their  identity  and  any  organizations  they  mentioned.  Four  of  the  subjects 
emphasized  the  need  for  anonymity  due  to  their  organizational  reliance  on  government 
evaluation.  Despite  one  subject’s  enthusiasm  to  participate  in  the  interview,  the  subject 
actually  refused  to  be  recorded.  In  this  case,  an  agreement  was  made  that  I  would  not 
record  and  would  only  take  notes.  The  need  to  do  this  hinged  on  the  fact  that  this 
particular  subject’s  input  was  valuable,  as  the  subject  was  a  systems  engineer  and  a  PRM 
SME  with  direct  PRM  experience  in  eight  prior  units.  It  was  also  the  case  for  six  other 
subjects  that  a  recording  could  not  be  obtained  because  their  schedule  only  permitted  an 
interview  within  the  work  setting  that  happened  to  be  at  a  sensitive  compartmented 
information  facility  where  recording  devices  were  not  allowed.  From  a  security 
perspective,  access  to  all  information  and  data  was  kept  secure  through  electronic 
password  protection  on  the  interviewer’s  personal  laptop. 

Investigative  Questions  Answered 

Emphasis  was  placed  on  ensuring  that  the  assumptions  made  about  PRM  were 
directly  addressed  by  the  five  foundational  questions  posed  in  Chapter  1.  The  five 
foundational  questions  served  as  a  baseline  for  reviewing  the  surrounding  literature  and 
laid  the  groundwork  for  creating  the  interview  question-set  used  in  the  research  study. 
Maintaining  focus  on  determining  the  underlying  phenomenon  at  hand  was  crucial 
because  the  problems  observed  by  the  author  are  inherently  ambiguous.  In  other  words, 
many  assumptions  could  be  made  about  the  observed  lack  of  PRM  consistency.  Take  for 
instance  viable  assumptions  that  the  observations  could  be  false,  biased,  or  exceptions  to 
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the  norm.  To  maintain  traceability,  the  results  of  the  study  will  be  explained  in  terms  of 
the  stated  assumption,  which  is  restated  below,  and  segmented  into  three  parts  with  their 
associated  questions  in  the  interview  set. 

Project  risk  management  processes  are  inconsistently  practiced  by  units  in  the  Air 
Force  because  of  ambiguous  direction,  cognitive  biases,  flawed  practices,  and 
inadequate  tools;  this  is  evidence  of  project  risk  mishandling  and  leads  to 
misinformed  decisions  at  the  Air  Force  enterprise  level. 

First  Stated  Assumption.  The  first  foundational  question  explores  the  first 

assumption  stated  in  chapter  one  that  “Project  risk  management  processes  are 

inconsistently  practiced  by  units  in  the  Air  Force. . .”  by  asking,  “Is  project  risk 

management  (PRM)  inconsistently  practiced  in  the  U.S.  Air  Force?”  To  better  facilitate 

conversation  in  an  interview  setting,  the  first  foundational  question  was  broken  out  into  a 

series  of  primary,  follow-up  probe,  and  Likert-scale  questions.  The  following  table  maps 

the  lineage  from  the  first  stated  assumption  and  foundational  question  to  the  interview 

questions. 


60 


Table  2 


Lineage  -  First  Assumption  and  Associated  Questions 


STATED  ASSUMPTION _ 

Project  risk  management  processes  are  inconsistently  practiced  by  units  in  the  Air 
Force... 

FOUNDATIONAL  QUESTION _ 

Is  project  risk  management  (PRM)  inconsistently  practiced  in  the  U.S.  Air  Force? 
INTERVIEW  QUESTIONS _ 

Flow  does  the  emphasis  on  PRM  processes  at  your  current  unit  compare  to  other  units 
you've  been  involved  with  or  are  aware  of? 

a.  In  how  many  different  Units  have  you  been  involved  with  the  PRM  process? 

b.  Did  you  encounter  or  know  of  any  units  that  either  lacked  or  had  little  regard  for 
PRM?  If  so,  why  do  you  think  that  is? 

c.  Were  the  differences  in  emphasis  you  describe  driven  by  necessity  differing  guidance, 
or  other  reasons? 

LIKERT-SCALE  QUESTION _ 

For  any  given  risk,  units  across  the  Air  Force  are  likely  to  arrive  at  the  same  assessment 
in  terms  of  the  probability  and  impact  that  would  be  reported. 

Strongly  Disagree  (6)  -  Disagree  (9)  -  Uncertain  (0)  -  Agree  (5)  -  Strongly  Agree  (0) 


Figure  1 

Likert-scale  -  Summary  of  First  Assumption 


Truncated  Likert-scale 
Question 

Disagreement  with  Assumptions 

Agreement  with  Assumptions 

Air  Force  PRM  is 

inconsistant 

< - i  THROUGH  20  SUBJECTS 

1  THROUGH  20  SUBJECTS - > 

Agreement/Disagrement: 
Strong  Agreement/Disagreement: 


Overall,  fifteen  out  of  twenty  subjects  agreed  that  PRM  was  not  consistently 
practiced  across  the  Air  Force.  A  detailed  look  at  the  responses  revealed  several 
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insightful  trends  associated  with  four  of  the  seven  attribute  ranges.  While  there  were 
strong  trends  discovered  with  experience,  rank,  mission  type,  and  budget,  there  were  no 
trends  evident  with  employer,  career  field,  or  product. 

The  strongest  trend  was  a  positive  relationship  between  experience  and  claims  of 
inconsistent  Air  Force  risk  management  processes.  In  this  case,  a  subject  was  considered 
experienced  after  two  or  more  previous  assignments  with  direct  exposure  to  their  unit’s 
PRM  process.  Twelve  of  the  fourteen  subjects  considered  experienced,  and  two  of  the 
subjects  considered  inexperienced,  explained  PRM  inconsistencies  throughout  the 
interview.  The  fourteen  subjects  who  explained  inconsistent  PRM  answered  concurrently 
on  the  Likert-scale  question  with  the  same  query.  It  was  interpreted  that  the  three 
subjects  with  little  to  no  experience  who  concurred  with  PRM  consistency  relied  on  the 
expectations  of  their  recent  initial  training  and  guidance  as  a  reflection  of  reality.  On  the 
other  hand,  the  twelve  subjects  with  greater  experience  explained  how  those  expectations 
did  not  materialize.  If  the  interpretation  is  correct,  then  there  may  be  aspects  of  the  Air 
Force’s  PRM  training  curriculum  that  deserve  assessment  for  applicability  and  relevance. 

Although  rank  is  a  different  attribute  than  experience,  the  two  tend  to  go  hand-in- 
hand.  It  is  possible  that  a  member  could  obtain  medium  to  higher  rank  without  having 
high  experience  in  PRM,  but  this  possibility  is  not  likely.  The  positive  relationship 
between  higher  rank  and  higher  experience  showed  true  for  all  subjects  in  the  study.  In 
other  words,  all  fourteen  members  with  high  experience  were  also  medium  to  high  rank. 
The  interpretation  of  the  relationship  between  higher  rank  and  concurrence  with  PRM 
inconsistencies  is  therefore  the  same  as  they  were  for  higher  rank.  That  said,  it  was 
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noteworthy  to  learn  that  the  higher-ranking  subjects  did  not  plan  to  use  their  authority  to 
implement  corrective  actions. 

Contrary  to  the  interpretations  and  assumptions  made  regarding  experience  and 
rank,  two  of  the  fourteen  subjects  with  high  experience  claimed  that  PRM  was  consistent. 
Interestingly,  these  two  highly  experienced  subjects  had  spent  their  careers  in  highly 
funded  space  programs,  with  the  majority  of  these  programs  being  greater  than  $100M. 
Furthermore,  nine  of  the  twelve  high  experienced  subjects  that  concurred  with 
inconsistent  PRM  processes  did  reference  consistency  between  highly  funded  Space 
programs  in  their  past  experiences.  These  patterns  seem  to  suggest  that  for  some  reason, 
consistent  PRM  processes  are  inherent  in  the  field  of  Space  acquisitions  and/or  highly 
funded  programs.  It  would  then  make  sense  that  two  out  of  three  of  the  less  experienced 
subjects  who  concurred  with  PRM  consistency  belonged  to  a  Space  acquisitions  unit. 
However,  the  pattern  is  also  concurrent  with  the  interpretation  that  low  experienced 
subjects  concur  due  to  expectations  from  recent  training  which  was  exclusively 
applicable  to  only  one  of  the  five  low  experienced  subjects.  The  two  low  experienced 
subjects  who  concurred  with  PRM  inconsistency  were  not  part  of  a  Space  acquisitions 
unit  nor  did  they  have  any  such  experience.  Overall,  the  data  suggested  that  there  is  a 
relationship  between  space  acquisitions  units  and  claims  that  PRM  is  consistent,  a 
positive  relationship  between  funding  and  claims  that  PRM  is  consistent,  and  a  negative 
relationship  between  experience  and  claims  that  PRM  is  consistent. 

The  two  most  frequent  inconsistencies  described  by  subjects  were  PRM  process 
emphasis  and  risk  handling  accountability.  Not  only  were  these  two  emphasis  areas 
described  as  inconsistent  across  units,  but  also  as  critical  areas  that  determined  the 
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success  of  the  unit’s  PRM  process.  The  majority  of  the  subjects  were  quick  to  point  out 
that  all  Air  Force  units  are  mandated  to  produce  risk  reporting  and  that  there  was  a  drastic 
difference  between  checking  a  box  and  intrinsic  action.  The  difference  driver  was  often 
described  to  be  leadership-buy-in.  As  it  takes  a  team  to  carry  out  a  PRM  process  on 
complex  systems,  individuals  will  get  discouraged  if  not  brought  together  as  a  team  to 
carry  out  the  endeavor  simultaneously.  Getting  beyond  the  box-checking  mentality  also 
meant  having  purpose,  as  the  purpose  of  having  a  PRM  process  is  to  identify  and  react  to 
project  risks  and  take  the  best  action.  If  there  is  no  action  towards  the  risks  identified,  or 
decision  of  non-action  for  that  matter,  then  there  is  no  purpose. 

Second  Stated  Assumption  -  Overview.  The  second  stated  assumption  attempts 
to  answer  why  PRM  processes  are  inconsistent  across  Air  Force  units.  Determining 
exactly  what  and  why  there  is  PRM  inconsistency  is  the  linchpin  of  the  study  and 
appropriately  received  the  greatest  research  emphasis.  According  to  the  author’s 
observations  directly  in  the  field  of  study,  PRM  inconsistency  can  be  attributed  to 
ambiguous  direction,  cognitive  biases,  flawed  practices,  and  inadequate  tools.  To 
promote  natural  discussion  for  an  interview  setting,  questions  where  designed  to 
simultaneously  probe  for  what  in  particular  was  inconsistent  and  why. 

Second  Stated  Assumption  -  Ambiguous  Direction.  Publications  are  to  the  Air  Force 
what  a  conductor  is  to  a  symphony.  The  Air  Force  is  large,  geographically  separated,  and 
highly  diversified  across  air,  space,  and  cyberspace  missions.  It  is  crucial  for  these 
diverse  domains  to  be  synchronized,  and  PRM  is  no  exception.  However,  funneling 
directions  and  guidance  from  the  headquarters  echelon  down  to  the  units  is  a  delicate 
balance  between  specificity  and  adaptability.  Air  Force  publications  must  synchronize 
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units  while  allowing  leeway  for  interpretation  and  tailoring  to  fit  unit  specific  missions. 
Based  on  the  author’s  experience  in  the  field  of  study  and  review  of  the  literature,  such 
balances  have  not  been  achieved,  and  as  a  result,  risk  management  directives  and 
guidance  seemed  ambiguous  and  contradictory. 

Table  3 

Lineage  -  Second  Assumption  and  Associated  Questions 

STATED  ASSUMPTION _ 

...because  of  ambiguous  direction... 

FOUNDATIONAL  QUESTIONS _ 

What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

What  contributes  to  Air  Force  PRM  inconsistencies?  <l> 

INTERVIEW  QUESTIONS _ 

What  publications  does  your  unit  use  and  why  do  you  think  that  it  was  selected? 

a.  Is  it  clear  which  publications  to  use? 

b.  Are  the  publications  used  understandable  and  specific? 

c.  Are  the  publications  used  adequate? 

LIKERT-SCALE  QUESTION _ 

There  is  clear  guidance  and  instruction  available  to  your  unit  on  how  to  carry  out  risk 
management. 

Strongly  Disagree  (12)  -  Disagree  (4)  -  Uncertain  (2)  -  Agree  (2)  -  Strongly  Agree  (0) 
Air  force  guidance  and  instruction  are  strictly  followed  in  your  unit. 

Strongly  Disagree  (12)  -  Disagree  (6)  -  Uncertain  (2)  -  Agree  (2)  -  Strongly  Agree  (0) 

Key: 

(1)  Questions  that  address  ‘why’  there  are  PRM  inconsistencies  are  highlighted  to  discern 
them  from  questions  that  address  ‘what’  PRM  inconsistencies  exist. 
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Figure  2 


Likert-scale  -  Summary  of  Second  Assumption 


Truncated  Likert-scale 
Question 


PRM  guidance/instruction  is 
not  clear 


PRM  guidance/instruction  is 
not  followed 


Disagreement  with  Assumptions 


1  THROUGH  20  SUBJECTS 


Agreement  with  Assumptions 


1  THROUGH  20  SUBJECTS  ■ 


Agreement/Disagrement: 
Strong  Agreement/Disagreement:  | 


In  terms  of  the  seven  attributes  recorded  for  each  subject,  trends  were  observed 
for  mission  type,  product  budget,  employer,  and  career  field.  No  significant  trends  were 
found  for  rank,  product,  or  experience.  The  most  notable  pattern  was  mission  type  and 
product  budget,  which,  as  mentioned  before,  are  not  mutually  exclusive.  There  was  a 
distinct  relationship  between  better  directives  and  space  acquisitions  units  found  within 
the  data.  There  was  also  a  positive  relationship  between  highly  funded  programs  and 
better  publications,  which  happens  to  be  indicative  of  space  acquisitions  programs. 
Although  risk  management  seems  to  be  a  long-lived  staple  of  space  missions,  funding 
seems  to  be  the  enabler  for  a  robust  risk  management  process.  A  strong  document 
management  process  was  common  between  all  programs  reported  as  having  consistent 
PRM  processes. 

The  author  discovered  that  document  management  involved  not  only  applying  the 
correct  documentation  and  references  to  the  acquisition  life  cycle,  but  also  tailoring 
higher-level  documents  to  the  unique  unit  level  mission.  Such  efforts  are  resource 
intensive  and  therefore  require  extra  funding.  Programs  that  did  not  have  strong 
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document  management  considered  the  endeavor  to  be  extraneous  to  their  primary  mission 
and  too  low  of  a  priority  to  adequately  fund.  All  twelve  subjects  that  had  experience 
working  in  a  highly  funded  space  acquisitions  units,  and  two  other  subjects  who  were 
working  in  a  highly  funded  aviation  acquisitions  unit,  reported  either  current  or  past 
experiences  with  clear  PRM  documentation  in  these  units.  The  commonality  in  the  cases 
of  sufficient  documentation  was  sufficient  funds  to  afford  document  management. 

It  could  be  argued  on  the  basis  of  the  data  from  this  research  study  that  a  majority 
of  Air  Force  units  reference  higher- level  PRM  guidance  with  little  to  no  success. 

Fourteen  subjects  who  either  had  current  or  past  experience  with  lower  funded 
acquisitions  units  reported  that  unit  level  PRM  guidance  was  not  available.  Twelve  of 
the  fourteen  subjects  who  did  not  have  unit  level  guidance  could  not  identify  the  higher- 
level  publication(s)  that  were  applicable  to  their  PRM  process,  and  as  a  result,  felt  that 
the  guidance  was  non-value  added.  Subjects  did  not  specifically  mention  contradiction  or 
confusion  about  what  higher-level  publications  to  use;  however,  eighteen  out  of  twenty 
subjects  described  ambiguity  and  vagueness  among  the  higher-level  publications.  Most 
notable  were  the  descriptions  from  the  five  PRM  SMEs  and  five  out  the  eight  systems 
engineers  who  directly  interpreted  higher-level  publications  in  efforts  to  create  tailored 
unit  level  PRM  guidance.  The  subjects  directly  involved  with  interpreting  higher  level 
PRM  guidance  reported  that  the  degree  of  vagueness  encountered  left  too  much  room  for 
interpretation. 

The  data  indicates  that  the  myriad  of  higher-level  PRM  publications  do  not 
achieve  a  unified  approach  to  PRM.  Per  the  interviews,  subjects  indicated  that  the 
publications  were  either  not  referenced  at  all  or  they  become  loosely  interpreted.  The 
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most  consistency  was  achieved  between  units  who  had  contractors  that  created  unit  level 
PRM  guidance.  In  these  cases,  contractor  interpretations  were  ultimately  influenced  by 
industry-best-standards.  Two  detached  separate  units  both  having  unit  level  PRM  and 
contractor  resources  had  similar  PRM  guidance  that  fit  high-level  PRM  criteria  but 
aligned  with  industry-best-standard. 

Second  Stated  Assumption  -  Cognitive  biases.  All  subjects  interviewed  explained  that 
there  existed  some  degree  of  cognitive  bias.  There  were  no  distinct  patterns  or  trends  that 
were  found  with  respect  to  any  of  the  seven  attributes.  That  said,  the  consistent  existence 
of  cognitive  bias  reflected  by  the  subjects  was  a  consistent  trend,  as  bias  has  the  potential 
to  produce  inconsistent  PRM  assessments  and  decisions.  The  research  study  bolstered 
the  earlier  claim  that  cognitive  biases  are  inherent  in  all  group  interactions,  but  that  the 
goal  is  not  to  just  identify  inevitabilities.  On  the  contrary,  the  effort  to  confirm  and 
pinpoint  the  different  biases  begins  the  understanding  of  how  to  control  for  and  mitigate 
biased  PRM  assessments  and  decisions. 
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Table  4 


Lineage  -  Third  Assumption  and  Associated  Questions 


STATED  ASSUMPTION 

...because  of  cognitive  biases... 

FOUNDATIONAL  QUESTIONS 

What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

What  contributes  to  Air  Force  PRM  inconsistencies?  (1) 

INTERVIEW  QUESTIONS 

What  collaboration  methods  do  you  use  (Ex:  Delphi  technique,  story  board,  meeting, 
email)  and  why  were  they  chosen? 

a.  Are  the  methods  you  mention  different  than  in  other  units  you  know  of?  Why  do  you 
think  there  is  a  difference? 

b.  Are  the  methods  you  mention  adequate  (i.e.  do  you  encounter  any  challenges  such  as 
bias,  strong  personalities,  or  group  think)? 

LIKERT-SCALE  QUESTION 

There  is  pressure  to  provide  positive  news  about  risks  when  reporting  to  supervisors  and 
leadership. 

Strongly  Disagree  (0)  -  Disagree  (1)  -  Uncertain  (2)  -  Agree  (9)  -  Strongly  Agree  (7) 

Group  dynamics  such  as  group  think  and  strong  personalities  influence  your  unit’s  risk 
assessments. 

Strongly  Disagree  ()  -  Disagree  (3)  -  Uncertain  (0)  -  Agree  (13)  -  Strongly  Agree  (4) 

Key: 

(1)  Questions  that  address  ‘why’  there  are  PRM  inconsistencies  are  highlighted  to  discern 
them  from  questions  that  address  ‘what’  PRM  inconsistencies  exist. 


Figure  3 

Likert-scale  -  Summary  Third  Assumption 


Truncated  Likert-scale 
Question 


There  is  pressure  to  report 
good  news  to  leadership 


Group  dynamics  influence 
risk  assessments 


Disagreement  with  Assumptions 


< . 1  THROUGH  20  SUBJECTS 


Agreement  with  Assumptions 


1  THROUGH  20  SUBJECTS  —  > 


Agreement/Disagrement: 
Strong  Agreement/Disagreement:  | 
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All  things  considered,  the  bias  described  as  skewing  PRM  reports  positively  for 
upper  leadership  and  management  generated  the  most  passionate  discussions  among  the 
research  subjects.  In  the  research  study,  two  reasons  were  provided  for  why  PRM  reports 
were  positively  skewed.  The  most  frequent  reason  given  by  13  of  the  20  subjects  was  to 
avoid  the  additional  oversight  brought  on  by  reported  high  risks.  It  would  make  sense  to 
provide  a  high-risk  project  with  more  resources  to  overcome  associated  challenges,  but 
instead,  the  expected  result  of  reporting  high  risk  seemed  to  be  the  opposite.  Resources 
would  be  required  to  provide  additional  reporting  to  upper  leadership  and  management 
instead  of  focusing  on  handling  the  high  risks  at  hand.  The  second  most  frequent  reason 
provided  by  1 1  subjects  was  to  preserve  reputation  of  both  stakeholders  and  the  project(s) 
at  hand.  For  a  project  in  a  risk  adverse  environment,  reporting  a  high  level  of  risk 
reduces  the  likelihood  for  further  personnel  and  funding  support.  For  stakeholders  in  the 
same  environment,  especially  government  systems  engineers  and  program  managers, 
reporting  a  high  level  of  risk  could  lead  to  a  chain  of  failures  and  ultimately  limit  the 
likelihood  for  career  progression. 

The  two  main  reasons  provided  by  subjects  in  the  research  study  were  contingent 
on  their  environments  being  risk  adverse.  In  hindsight,  it  would  have  been  insightful  to 
directly  ask  the  subjects  about  their  unit’s  tolerance  to  risk.  However,  four  subjects 
strongly  alluded  to  a  risk  adverse  environment  and  the  two  main  reasons  provided  for 
avoiding  reporting  high  risk  indicate  a  strong  possibility  for  a  trend  of  risk  adversity  in 
the  Air  Force. 

The  second  most  emphasized  problem  pertaining  to  cognitive  bias  was  a 
combination  of  groupthink  and  strong-voice.  Seven  subjects  described  a  common 
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scenario  where  someone  in  a  collaborative  environment  with  a  strong  voice  or 
dominating  personality  would  trigger  groupthink.  Two  subjects  alluded  to  PRM  topics  as 
susceptible  to  strong  opinions  and  groupthink  because  the  endeavor  was  inherently 
assumption  based.  In  other  words,  PRM  was  likened  to  the  abstract  realm  of  future 
predictions  where  no  opinions  could  be  grounded,  but  merely  swayed  by  the 
temperament  of  any  given  person  or  group. 

The  third  cognitive  bias,  mentioned  by  4  subjects  was  positivity  bias.  Positivity 
bias  could  be  possibly  tied  into  the  tendency  to  report  good  news  to  upper  leadership  and 
management  as  having  the  same  effect.  Subjects  explained  the  tendency  to  underestimate 
the  probability  and/or  impact  of  potential  risks.  Similarly,  subjects  also  reported 
tendencies  of  stakeholders  to  underestimate  schedule  length  and  costs  and  overestimate 
technical  and  human  capabilities. 

The  cognitive  biases  explained  by  all  twenty  subjects  are  not  specific  to  only 
groups  wrestling  with  PRM;  rather,  they  are  inherent  in  all  group  collaboration.  The 
purpose  for  exploring  the  cognitive  biases  experienced  by  the  subjects  is  to  understand 
their  particular  effect  on  PRM  assessments  and  decisions.  Characterizing  the  full 
dynamic  of  cognitive  biases  in  the  PRM  setting  will  shape  the  recommendations  for 
corrective  actions  to  be  discussed  in  chapter  five  of  this  thesis. 

Second  Stated  Assumption  -  Flawed  Practices.  The  assumption  that  Air  Force 
units  use  flawed  techniques  and  practices  was  based  on  the  author’s  observations  and 
review  of  the  literature.  The  implication  with  using  flawed  practices  does  not  directly 
result  in  inconsistent  risk  level  assessment  margins  of  error.  In  theory,  if  all  Air  Force 
units  were  consistently  using  the  same  flawed  practices,  there  would  be  consistent  risk 
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level  assessment  margins  of  error.  Although  in  a  few  interviews  there  were  indications  of 
inconsistent  flawed  practices  leading  to  consistent  margins  of  error,  consistency  cannot 
be  decoupled  from  accuracy.  Consistency  is  achieved  by  converging  on  accurate  risk 
assessments.  As  such,  evidence  of  inaccurate  risk  level  assessment  error  is  an  indication 
of  inconsistency  and  can  be  partially  attributed  to  flawed  practices  where  they  exist. 
Flawed  PRM  practices  must  be  identified  and  understood  in  order  to  take  corrective 
action. 
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Table  5 


Lineage  -  Fourth  Assumption  and  Associated  Questions 


STATED  ASSUMPTION 

...because  of  flawed  practices. . . 

FOUNDATIONAL  QUESTIONS 

What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

What  contributes  to  Air  Force  PRM  inconsistencies?  (1) 

INTERVIEW  QUESTIONS 

What  techniques  does  your  unit  use  (Ex:  risk  matrix,  montecarlo)?  Why  do  you  think  these 
specific  techniques  are  used? 

a.  Do  these  techniques  differ  from  other  units  and  what  do  you  think  is  the  cause? 

b.  Do  you  think  these  techniques  are  adequate?  Why  or  why  not? 

What  risk  identification  methods  (Ex:  email,  meetings,  delphi)  are  used  in  your  unit  and 
why  do  you  think  these  specific  techniques  are  used? 

a.  Are  there  differences  between  the  methods  used  in  your  unit  versus  other  units  that  you 
know  of  and  if  so,  what  do  you  think  is  the  cause? 

b.  Do  you  think  these  methods  adequately  identify  notable  risks? 

What  are  the  analysis  techniques  used  to  assess  likelihood  and  impact  and  why  do  you 
think  these  specific  techniques  are  used? 

a.  Are  there  differences  between  the  techniques  used  in  your  unit  versus  other  units  that 
you  know  of  and  if  so,  what  do  you  think  is  the  cause? 

b.  Do  you  think  these  techniques  produce  accurate  assessments? 

LIKERT-SCALE  QUESTION 

Different  units  in  the  Air  Force  use  the  same  methods  and  techniques  to  assess  project 
risks. 

Strongly  Disagree  (8)  -  Disagree  (8)  -  Uncertain  (1)  -  Agree  (3)  -  Strongly  Agree  (0) 

The  techniques  and  methods  used  by  your  unit  to  carry  out  PRM  processes  are  adequate 
and  are  equally  as  effective  as  industry  best-practices. 

Strongly  Disagree  (14)  -  Disagree  (3)  -  Uncertain  (1)  -  Agree  (2)  -  Strongly  Agree  (0) 

Key: 

(1)  Questions  that  address  ‘why’  there  are  PRM  inconsistencies  are  highlighted  to  discern 
them  from  questions  that  address  ‘what’  PRM  inconsistencies  exist. 
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Figure  4 


Likert-scale  -  Summary  of  Fourth  Assumption 


Truncated  Likert-scale 
Question 


Different 

methods/techniques  are 
used  to  ID  risks 


Different 

methods/techniques  are 
used  to  assess  risks 


Tecnniques/methods  lag 
behind  industry  best 
standards 


Disagreement  with  Assumptions 


1  THROUGH  20  SUBJECTS 


Agreement  with  Assumptions 


1  THROUGH  20  SUBJECTS  - 


Agreement/Disagrement: 

Strong  Agreement/Disagreement:  | 

In  terms  of  the  seven  attribute  areas,  there  were  trends  that  were  associated  with 
mission  type,  employer,  career  field,  and  product  budget.  However,  closer  analysis  of  the 
data  pointed  to  the  career  field  and  employer  attributes  as  the  drivers  for  the  observed 
pattern  while  the  other  attributes  were  merely  shared  by  the  same  subjects.  While  all 
subjects  claimed  to  some  degree  that  Air  Force  PRM  practices  were  not  as  effective  as 
industry  best  standards,  PRM  SMEs  and  six  out  of  the  eight  systems  engineers  were  able 
to  explicitly  explain  why  the  Air  Force  is  lagging.  The  data  suggests  that  the  PRM  SME 
and  systems  engineering  career  fields  afforded  the  subjects  a  more  focused  role  with 
PRM.  Contractors  had  the  most  insight  into  the  comparison  between  industry  best- 
standard  practices  and  Air  Force  practices,  as  their  respective  companies  were  an  integral 
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part  of  industry  and  defining  what  was  considered  best-standard  practices.  That  said, 
there  was  a  positive  association  between  a  greater  understanding  of  PRM  practices  and 
greater  claims  that  Air  Force  PRM  practices  were  lagging. 

The  most  notable  Air  Force  practice  articulated  by  the  subjects  during  the 
interviews  was  the  five-by-five  risk  matrix,  also  termed  a  risk  cube.  The  risk  matrix  was 
consistently  described  as  prolific  and  central  to  Air  Force  PRM.  It  was  also  notable  that 
the  risk  matrix  was  the  only  Air  Force  practice  mentioned.  On  the  other  hand,  the 
contractor  subjects  were  able  to  identify  a  multitude  of  techniques  used  in  the  private 
sector  considered  necessary  to  handle  PRM  for  complex  projects  and  programs.  The 
majority  of  the  subjects  criticized  that  the  risk  matrix  was  non-deterministic  and 
subjective.  The  contractor  PRM  SMEs  and  systems  engineers  who  had  in-depth  PRM 
knowledge  further  explained  the  inability  of  the  risk  matrix  to  account  for  compounded 
risks,  and  that  consistency  cannot  be  decoupled  from  accuracy.  Indeed,  these  subjects 
noted  that  consistency  is  achieved  by  converging  on  accurate  risk  assessments. 

Second  Stated  Assumption  -  Inadequate  Tools.  The  implication  with 
inadequate  tools  is  analogous  to  flawed  practices;  proficiency  must  be  achieved  in  order 
to  consistently  converge  on  accurate  risk  assessments.  According  to  the  subjects  familiar 
with  industry  best-standard  PRM  tools  in  the  private  sector,  tools  can  promote  rigor, 
accountability,  new  processes  and  techniques  through  behavior-shaping  constraints.  The 
patterns  deduced  from  the  interview  data  in  terms  of  the  seven  attribute  areas  were  also 
similar  to  that  of  flawed  practices. 
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Table  6 


Lineage  -  Fifth  Assumption  and  Associated  Questions 


STATED  ASSUMPTION 

. .  .because  of  inadequate  tools. . . 

FOUNDATIONAL  QUESTIONS 

What  parts  of  the  PRM  processes  across  the  Air  Force  tend  to  be  inconsistent? 

What  contributes  to  Air  Force  PRM  inconsistencies?  (1) 

INTERVIEW  QUESTIONS 

What  tools  does  your  unit  use  to  implement  the  PRM  process?  Why  do  you  think  these 
specific  tools  are  used? 

a.  Flow  are  these  tools  different  than  other  units  and  what  do  you  think  is  the  cause? 

b.  Are  these  tools  available  to  the  right  personnel  (Ex:  Active  risk  manager,  MSOffice)? 

Why  do  you  think  these  specific  tools  are  available? 

c.  Are  the  tools  you  mentioned  adequate?  Why  or  why  not? 

d.  On  which  portion  of  the  risk  management  process  are  the  tools  you  mentioned  actually 
used?  Why  or  why  not  certain  parts? 

LIKERT-SCALE  QUESTION 

The  tools  used  by  your  unit  for  PRM  processes  at  the  CGO/mid-management  level  are 
adequate  and  are  equally  as  effective  as  industry-best-practices. 

Strongly  Disagree  (16)  -  Disagree  (3)  -  Uncertain  (1)  -  Agree  (0)  -  Strongly  Agree  (0) 

Key: 

(1)  Questions  that  address  ‘why’  there  are  PRM  inconsistencies  are  highlighted  to  discern 
them  from  questions  that  address  ‘what’  PRM  inconsistencies  exist. 


Figure  5 

Likert-scale  -  Summary  of  Fifth  Assumption 


Truncated  Likert-scale 

Disagreement  with  Assumptions 

Agreement  with  Assumptions 

Question 

Tools  lag  behind  industry- 
best-standards 

< - 1  THROUGH  20  SUBJECTS 

1  THROUGH  20  SUBJECTS - > 

Agreement/Disagrement: 

Strong  Agreement/Disagreement:  | 

The  research  study  revealed  trends  that  were  related  to  mission  type,  employer, 
career  field,  and  product  budget.  However,  closer  analysis  of  the  data  pointed  to  the 
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career  field  and  employer  attributes  as  the  drivers  for  the  observed  pattern.  While  all 
subjects  claimed  to  some  degree  that  Air  Force  PRM  tools  were  not  as  effective  as 
industry  best  standards,  PRM  SMEs  and  four  out  of  the  eight  systems  engineers  were 
able  to  explicitly  explain  why  the  Air  Force  is  lagging.  Interpreting  the  data  revealed  that 
the  PRM  SME  and  systems  engineering  career  fields  afforded  the  subjects  a  more 
focused  role  with  PRM,  and  as  a  result,  better  insight  into  the  inadequacies.  Contractors 
had  the  most  insight  into  the  comparison  between  industry  best-standard  tools  used  in  the 
private  sector  and  Air  Force  tools,  as  their  respective  companies  were  an  integral  part  of 
industry  and  defining  what  was  considered  best-standard  practices.  That  said,  there  was  a 
positive  association  between  a  greater  understanding  of  PRM  tools  and  greater  claims 
that  Air  Force  PRM  practices  were  lagging. 

The  Air  Force’s  designated  tool  for  PRM  was  confirmed  by  eighteen  out  of 
twenty  subjects  to  be  Active  Risk  Manager  (ARM)  as  predicted  earlier  from  the  author’s 
observations  and  review  of  the  literature.  ARM  was  found  to  only  be  available  to  six  of 
the  twenty  subjects,  two  of  the  eight  systems  engineers  and  four  of  five  contractor  SMEs. 
Only  one  systems  engineer  and  two  contractor  SMEs  had  the  aptitude  to  use  ARM.  Of 
the  other  fourteen  subjects  that  had  no  direct  experience  with  ARM,  only  eleven  were 
aware  of  tool’s  existence.  It  was  assumed  that  the  other  two  subjects  were  not  aware  of 
the  tool  due  to  their  low  experience.  It  is  important  to  remember  the  surrounding 
circumstances  to  comprehend  the  problem  at  hand.  Given  the  criteria,  all  subjects 
interviewed  should  have  had  an  integral  role  in  the  handling  of  their  program’s  risks. 
According  to  the  contractor  SMEs  and  industry  best  standards,  PRM  stakeholders  such  as 
the  subjects  interviewed  should  ideally  all  collaborate  on  a  tool  that  is  dedicated  to  PRM. 
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It  became  evident  through  the  research  study  that  the  root  cause  of  why  ARM  was 
not  used  by  stakeholders  was  threefold.  First,  it  was  too  intricate  and  complicated  to  use 
as  those  subjects  who  used  it  were  dedicated  PRM  SMEs  whose  primary  responsibility 
was  PRM  while  those  who  did  not  use  it  had  multiple  other  responsibilities.  For  instance, 
in  addition  to  PRM,  a  program  manager  has  to  contend  with  balancing  cost,  schedule, 
scope,  and  resources  for  their  project(s).  A  systems  engineer,  by  definition,  is 
responsible  for  eleven  disciplines  other  than  just  PRM,  which  can  often  simultaneously 
include  the  management  of  requirements,  testing  and  evaluation,  human  factors  and 
architecture  integration.  Second,  the  subjects  who  were  experienced  with  ARM  claimed 
that  when  the  tool  was  used  to  its  fullest  extent,  that  it  was  only  proficient  at  reporting  as 
opposed  to  the  actual  management  and  handling  of  the  identified  risks.  Third,  licensing 
for  ARM  was  said  to  be  too  expensive  to  provide  to  all  stakeholders. 

The  research  study  confirmed  the  author’s  observations  and  review  of  the 
surrounding  literature  regarding  PRM  tools  in  the  Air  Force.  The  only  dedicated  tool  was 
ARM  and  it  is  inadequate  in  terms  of  usability,  suitability,  and  availability.  The 
inadequacy  of  ARM  left  PRM  stakeholders  using  generic  tools  provided  by  Microsoft 
Office  to  manage  risk.  Microsoft  Office  tools  are  capable,  usable,  and  available  for  a 
wide  range  of  purposes  in  the  Air  Force,  but  are  not  specifically  designed  for  PRM. 
Compared  to  PRM  tools  that  are  considered  to  be  the  industry-best-standard,  Microsoft 
Office  tools  lack  automation,  digital  form  templates  that  standardize  techniques,  and  a 
platform  for  collaboration. 
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Summary  -  Mission  Impacts 

The  research  effort  provided  insight  that  grounded  the  author’s  stated  assumptions 
and  reinforced  the  information  found  in  the  surrounding  literature.  Validation  in  this 
manner  was  key  in  identifying  the  deficiencies  surrounding  Air  Force  PRM,  as  it  runs 
counter  to  the  incentives  of  those  who  are  usually  aware  of  them.  Furthermore,  the  key  to 
unraveling  the  complex  combination  of  contributors  to  Air  Force  PRM  inconsistency  was 
held  deep  within  the  knowledge  of  the  PRM  subjects  interviewed.  Not  only  was  valuable 
insight  gained,  but  cross-referencing  a  diverse  set  of  perspectives  for  trends  and  patterns 
provided  comprehensive  insight  to  the  underlying  phenomenon  behind  PRM 
inconsistencies. 

In  efforts  to  invoke  change,  a  shift  in  focus  from  characterizing  AF  PRM 
deficiencies  to  mission  impacts  will  ensure  effective  changes  are  made.  The  third  and 
last  stated  assumption  claimed  that  not  only  did  PRM  inconsistencies  evidence  risk 
mishandling  at  the  program  level,  it  also  led  to  misinformed  decisions  at  the  Air  Force 
enterprise  level.  Take  for  instance  this  scenario.  Unit  A  carries  out  an  accurate  risk 
management  assessment  on  risk  X,  and  unit  B  carries  out  a  positively  skewed  risk 
assessment  on  the  same  risk  X.  Not  only  is  unit  B  postured  to  underreact  to  the  risk,  but 
on  an  enterprise  level,  the  erroneous  assumption  will  be  made  that  unit  B  is  a  less  risky 
investment  for  further  funding  and  resources. 

In  hindsight,  there  was  a  lack  of  emphasis  in  the  interview  set  on  discussing 
mission  impacts  resulting  from  deficient  and  inconsistent  Air  Force  PRM.  There  were 
two  questions,  one  addressing  the  impacts  at  the  unit  level  and  the  other  addressing  the 
enterprise  level.  The  two  questions  occurred  at  the  end  of  the  lengthy  interview  and  all 
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twenty  subjects  apathetically  agreed  with  the  obvious;  bad  things  happen  if  PRM  is  not 
proficiently  and  consistently  carried  out  across  Air  Force  units.  Although  misinformed 
decisions  at  both  the  unit  and  enterprise  level  can  lead  to  an  onslaught  of  complications, 
the  objective  is  as  simple  as  the  answers  provided.  Measures  need  to  be  taken  towards 
improving  Air  Force  PRM  proficiency  and  consistency  by  focusing  on  the  four 
problematic  areas  identified:  publications,  biases,  practices,  and  tools. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

The  motivation  to  scrutinize  Air  Force  PRM  stemmed  from  the  author’s 
experiences  directly  in  the  field  of  study.  While  many  were  observed  to  give  up  on  PRM 
and  resort  to  “box  checking,”  the  author  believed  in  the  potential  of  PRM  to  achieve 
positive  mission  effects.  Stated  more  succinctly,  the  author  believed  that  risks  could  be 
identified  and  assessed  to  inform  decisions  and  generate  action  as  opposed  to  just  serve  as 
another  meeting  topic.  Follow-through  beyond  reporting  is  also  applicable  to  the  efforts 
of  the  research  study  conducted. 

Given  the  author’s  observations  directly  in  the  field  of  study  and  the  review  of  the 
surrounding  literature  grounded  in  the  results  of  the  research  study,  a  characterization  of 
the  Air  Force  PRM  environment  will  be  articulated.  It  is  necessary  to  understand  the 
current  state  to  determine  how  to  achieve  the  desired  state.  In  the  case  of  the  research 
study,  an  explanation  of  what  and  why  AF  PRM  inconsistencies  exists  will  be  provided 
along  with  an  understanding  of  the  resulting  mission  impacts.  An  examination  of  the 
research  results  will  reveal  additional  areas  for  further  study.  In  the  spirit  of  follow- 
through,  an  examination  of  any  Air  Force  PRM  deficiencies  found  will  serve  as  a  launch 
pad  for  further  action. 

Conclusions  of  Research 

As  predicted  by  the  author,  the  research  conducted  revealed  inconsistencies  and 
deficiencies  within  AF  PRM.  The  fourteen  subjects  who  reported  AF  PRM  to  be 
inconsistent  also  reported  that  publications,  biases,  practices,  and  tools  were  all  factors 
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that  contributed  to  inconsistent  risk  management  to  some  degree.  Other  contributors 
were  introduced  in  the  study  but  did  not  stand  out  as  significant  if  they  were  only 
mentioned  two  or  less  times  and/or  considered  to  be  a  sub-factor  to  one  of  the  four 
assumed  contributors.  However,  lack  of  education  and  training  was  mentioned  thirteen 
times  and  stands  out  independently  as  another  contributing  factor  to  AF  PRM 
inconsistency. 

Comparing  and  contrasting  the  interview  data  in  terms  of  the  subject’s  salient 
attributes  revealed  patterns  about  the  underlying  phenomenon  causing  PRM 
inconsistencies  and  deficiencies.  Analysis  of  the  seven  attribute  areas  revealed  mission 
type  as  a  source  for  the  most  trends  discovered.  Overall,  space  mission  units 
demonstrated  more  PRM  emphasis  and  proficiency  than  air  and  cyberspace  mission  units. 
Mainly,  the  criticality  of  space  missions  and  the  catastrophic  consequences  of  failure 
necessitates  higher  PRM  effectiveness.  The  requirement  for  PRM  effectiveness  drove 
trends  in  other  attribute  areas  such  as  the  availability  of  funding  which  enabled  the  onset 
of  contractors  who  tend  to  have  focused  expertise  in  PRM.  In  this  case,  the  mission  type 
attribute  drove  the  product  budget,  employer,  and  career  field  attribute  areas.  The 
research  study  has  demonstrated  that  space  units  can  be  used  by  the  Air  Force  as  a  model 
to  improve  PRM  proficiency. 

A  macro  perspective  of  the  research  study  showed  inconsistencies  across  the  Air 
Force  contingent  on  five  of  the  seven  attribute  areas  explored.  In  regards  to  the  mission 
type  attribute,  PRM  proficiency  inconsistencies  were  found  between  aviation, 
cyberspace,  and  space  units.  The  conclusion  was  made  that  a  high  need  for  PRM  in 
space  units  drove  higher  funding.  As  such,  funding  did  not  drive  the  inconsistency; 
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rather,  funding  enabled  the  higher  PRM  proficiency  through  the  hiring  of  contractor 
SMEs  familiar  with  industry  best  practices.  Inconsistency  between  different  levels  of 
rank  were  concluded  to  be  driven  by  experience  and  was  therefore  thrown  out  as  a  factor. 
The  trends  associated  with  the  experience  attribute  helped  explain  the  outlying  data 
provided  by  two  inexperienced  subjects  not  in  a  space  unit  who  claimed  that  there  were 
no  PRM  deficiencies  or  inconsistencies.  Analysis  of  the  data  led  to  the  understanding 
that  a  more  accurate  interpretation  of  PRM,  one  with  deficiencies  and  inconsistencies, 
was  gained  through  experience.  The  table  below  provides  an  outline  of  the  interpreted 
interview  data  in  terms  of  the  seven  attribute  areas. 
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Table  7 


Summary  of  Attribute  Trends  and  Obsen’ations 


ATTRIBUTE 

RANGE  &  CONCLUSIONS 

Mission  Type 

Air 

Cyberspace 

Space 

Aviation  and  cyber  units  had  a  low  emphasis  on  PRM  compared  to 
Space  units  resulting  from  the  need  for  high  reliability.  Aviation  units 
relied  on  operational  risk  management  for  mission  reliability. 

Employer 

Active  Duty 

Gov’t  Civilian 

Contractor 

Active  duty  and  gov ’t  civilian  personnel  were  similar  in  terms  of  their 
knowledge  and  implementation  of  PRM.  Gov ’t  civilians  assigned  to 
PRM  &  contractors  were  highly  adept  to  PRM. 

Rank  (1) 

Low 

Medium 

High 

The  trends  and  patterns  found  ran  concurrent  with  experience,  which 
was  determined  as  the  true  diver  of  higher  PRM  situational  awareness 
and  competence. 

Career  Field 

Systems  Engineer 

Program  Manager 

PRM  SME 

Systems  Engineers  &  program  managers  were  similar  in  terms  of  their 
knowledge  &  implementation  of  PRM  while  PRM  SMEs  ( contractors ) 
were  highly  adept  to  PRM  (similar  to  the  ‘employer  ’  attribute). 

Product 

New  Tech 

Support  Equipment 

Weapon  System 

There  were  no  specific  patterns  or  trends  found. 

Product  budget 

<$10M 

$10M  <  $100M 

>  $100M 

Aviation  and  cyber  units  were  funded  less  than  $100M  and  had  a  low 
emphasis  on  PRM  compared  to  Space  units  funded  over  $100M.  This 
attribute  is  concurrent  with  the  ‘mission  type  ’  attribute. 

Experience  (2) 

High  Low 

Subjects  with  higher  experience  reported  more  inconsistencies  and 
deficiencies  than  subjects  with  lower  experience.  It  was  deduced  that  a 
more  accurate  interpretation  of  PRM  was  gained  through  experience. 
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Significance  of  Research 

The  objective  of  the  research  study  was  to  reveal  the  phenomenon  responsible  for 
inconsistent  Air  Force  PRM  so  that  corrective  actions  can  be  focused  appropriately. 
Though  the  author’s  explanation  of  the  phenomenon  came  directly  from  observations  in 
the  field  of  study  and  from  over  nine  years  of  experience,  further  substantiation  was 
needed.  Assumptions  limited  to  just  the  author’s  perspective  could  be  rebutted  in 
multiple  ways.  It  could  be  argued  that  the  author’s  scope  of  experience  is  not 
representative  of  the  whole  U.S.  Air  Force.  On  the  other  hand,  the  research  study  was 
comprehensive  in  that  it  tapped  into  the  knowledge  of  twenty  subjects  who  represented 
multiple  Air  Force  perspectives.  That  said,  the  research  study  bolstered  the  author’s 
assumptions  and  introduced  new  information  regarding  the  deficiencies  surrounding  Air 
Force  PRM.  The  results  from  the  research  study  can  be  used  as  a  launch  point  for  further 
research  and  corrective  actions. 

Recommendations  for  Action 

The  results  from  the  research  study  point  to  space  acquisitions  units  as  the  most 
proficient  in  PRM.  The  trends  show  that  higher  funding  for  contractor  PRM  SMEs  and 
their  knowledge  of  industry  best  practices  have  the  tendency  to  promote  effective  PRM 
practices.  Interjecting  more  contractors  with  industry  knowledge  is  an  example  that  can 
be  followed  directly,  but  the  focus  should  be  on  aligning  Air  Force  practices  with 
industry  best  standards.  Despite  units  that  had  an  abundance  of  industry  savvy 
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contractors,  PRM  was  short  of  full  potential  due  to  lack  of  sufficient  tools  and 
community  buy-in. 

A  simple  and  affordable  computer-based  tool  designed  to  manage  all  aspects  of 
PRM  should  be  made  available  to  all  PRM  stakeholders.  Such  a  tool  would  most 
effectively  address  the  deficiencies  outlined  in  the  research  study  if  it  satisfied  the 
following  requirements: 

•  Shall  interface  with  Microsoft  Outlook  calendars  to  send  auto  reminders  about  action 
item  due  dates  and  descriptions  (Similar  to  SharePoint) 

•  Shall  interface  with  Microsoft  Outlook  email  to  provide  updates  and  solicit  for  risk 
identification  and  input  (Delphi  method) 

•  Shall  be  virtually  accessible  simultaneously  as  to  promote  real  time  collaboration 

•  Shall  force  a  standard  reporting  format  consistent  to  industry-best-standard  accessible 
at  any  time  to  account  holders  with  access  rights 

•  Shall  be  able  to  produce  reports  in  Microsoft  PowerPoint 

•  Shall  be  able  to  produce  reports  in  portable  document  format  (PDF) 

•  Shall  be  able  to  produce  reports  in  Microsoft  Excel 

•  Shall  force  all  steps  of  the  PRM  process 

•  Shall  virtually  store  identified  risks  and  all  salient  risk  information 

•  Shall  interface  with  Microsoft  Outlook  to  solicit  for  risk  assessment  input  (Delphi 
method) 
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•  Shall  track  risk  assessments  to  include,  but  not  limited  to,  trigger  criteria,  probability, 
impact,  and  the  compounded  effects  of  multiple  risk  occurrence  (potential  effects  of 
mutually  inclusive  risks) 

•  Shall  automatically  track  and  update  reports  when  risk  is  reduced  through  the 
completion  of  risk  handling  action  items 

The  endeavor  of  implementing  a  PRM  tool,  or  any  tool  for  that  matter,  is  a 
daunting  and  cumbersome  task.  There  would  be  many  aspects  to  consider  such  as 
enterprise-wide  installment,  compatibility,  license  accountability,  software  integration, 
process  integration,  funding,  and  security  measures,  to  name  a  few.  There  is  also  a  high 
potential  for  conflicts  with  the  current  contracts  between  the  Air  Force  and  ARM,  not 
only  from  a  proprietary  perspective  but  also  in  terms  of  redundancy.  There  would  most 
likely  be  confusion  as  to  which  tool  to  use  and  how  to  transition.  Planning  would  have  to 
account  for  when  ARM  would  terminate  and  the  length  of  time  the  new  tool  would 
overlap.  Coordination  efforts  would  need  to  consider  the  transfer  of  information  from 
ARM  to  the  new  tool.  Or,  if  both  tools  were  kept,  funding  would  have  to  be  justified  in 
an  already  financially  strained  environment.  Much  of  the  pain  involved  with  introducing 
a  new  tool  could  be  avoided  by  enhancing  the  already  approved  and  existing  ARM  tool 
with  add-in  lite  version.  Specifically,  each  full  version  license  should  come  with  ten  lite 
versions  that  can  be  distributed  to  non-SME  stakeholders.  A  lite  version  would  be 
suitable,  available,  and  useable.  Below  is  a  business-process-model-and-notation 
(BPMN)  diagram  that  graphically  represents  the  process  flow  for  how  such  a  tool  fulfill 
the  previously  listed  requirements. 
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Figure  6 
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Recommendations  for  Future  Research 


When  implementing  corrective  action  and  process  change,  it  is  important  to 
consider  the  human  element.  Further  research  should  investigate  the  next  PRM  tool  or 
ARM  add-in  from  a  human  factors  perspective  to  ensure  the  “ilities”  are  in  line  with  what 
is  needed  and  what  will  be  used.  Doing  so  will  ensure  the  best  possible  chance  for 
leadership  and  unit  buy-in.  In  the  interviews  and  the  author’s  experience,  buy-in  is 
critical  to  the  success  of  a  new  process  and/or  tool  as  change  tends  to  be  inevitably 
challenging  and  inherently  resisted. 

Training  could  act  as  catalyst  to  the  paradigm  shift  that  is  needed  in  Air  Force 
PRM.  Not  only  should  training  correspond  to  the  latest  techniques  in  the  field,  it  is  also 
an  opportunity  to  introduce  and  stay  current  on  industry  best  standards.  In  addition  to 
opportunities  to  affect  change,  training  was  recognized  as  a  weak  area  in  the  research 
study.  It  would  be  advantageous  to  know  what  is  currently  being  taught  and  to  what 
extent. 

In  addition  improving  training,  the  possibility  of  centralizing  PRM  change 
implementation  to  a  “schoolhouse”  should  be  explored.  Such  a  school  house  could  be 
virtual  or  physically  located  where  PRM  training  takes  place  and  would  act  as  the 
authority  for  pushing  software  updates  to  the  new  PRM  tool  as  industry  best  standards 
change  and  improve. 

Aside  from  industry  best  standards  as  a  source  for  PRM  practices  and  training, 
other  service  branches  should  be  researched  for  examples  that  work,  or  do  not  work,  in 
the  unique  military  setting.  Aggregating  risk  data  across  the  entire  DoD  would  form  a 
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higher  macro  level  than  previously  discussed  and  work  well  to  inform  crucial  decisions 
made  at  the  presidential  level. 

Summary 

PRM  is  a  mainstay  in  Air  Force  acquisitions,  as  missions  become  ever  more 
challenging,  so  does  the  complexity  of  systems  and  the  associated  risk.  Air  Force 
acquisitions  teams  and  personnel  who  rely  on  intuition  and  reactionary  measures  will 
falter;  an  effective  and  comprehensive  PRM  plan  is  key  to  project  success.  The  literature 
review  outlined  a  series  of  historical  events  that  prove  the  worth  of  PRM.  Catastrophe 
has  ensued  when  PRM  was  neglected  and  avoided  when  PRM  was  effectively  practiced. 

Inspired  by  the  potential  of  PRM  in  Air  Force  acquisitions,  the  author  has 
scrutinized  PRM  effectiveness  directly  in  the  field  of  study  for  over  nine  years.  From  the 
author’s  vantage  point,  there  was  evidence  of  an  underlying  phenomenon  impeding  the 
full  potential  of  PRM  in  Air  Force  acquisitions.  The  scope  of  this  thesis  was  therefore 
bom  out  of  the  author’s  observations.  Specifically,  the  thesis  effort  sought  to  determine 
if  there  is  indeed  inconsistency  between  Air  Force  PRM  processes,  and  if  confirmed, 
what  the  cause  is,  and  the  ensuing  impact.  The  literature  reviewed  supported  the  author’s 
assumptions  that  Air  Force  PRM  inconsistencies  could  be  attributed  to  ambiguous 
publications,  cognitive  biases,  flawed  practices,  and  inadequate  tools.  Supporting 
evidence  would  serve  as  an  indication  of  risk  mishandling  at  the  Air  Force  unit  level  and 
misinformed  decisions  at  the  Air  Force  enterprise  levels. 

To  build  the  rapport  necessary  to  initiate  corrective  action,  further  validation  was 
pursued.  The  author’s  assumptions  and  findings  in  the  surrounding  literature  were 
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validated  through  a  qualitative  grounded  theory  research  study.  The  study  was  designed 
to  interview  a  pool  of  twenty  subjects  who  represented  a  wide  slice  of  Air  Force 
acquisitions  personnel.  Ultimately,  the  study  provided  insight  to  the  underlying 
phenomenon  causing  inconsistent  and  deficient  Air  Force  PRM. 

In  addition  to  addressing  the  initial  assumptions,  the  study  revealed  trends  and 
patterns  in  the  data  to  provide  new  insight  into  what  causes  and  enables  both  good  and 
bad  Air  Force  PRM.  The  information  from  the  research  study  can  be  used  to  focus 
corrective  efforts  and  changes  on  the  sources  of  the  problems  identified.  Air  Force 
leaders  and  units  responsible  for  program  execution,  and  more  importantly,  the  success  of 
warfighters  and  our  Nation’s  military  objectives,  must  continue  to  measure  and  improve 
project  risk  management  rigor. 
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Appendix  A 


Thesis  Interview 

Thesis:  Air  Force  Project  Risk  Management  (PRM)  Inconsistency  and  Potential 
Solutions 

Study:  Investigation  of  Air  Force  Project  Risk  Management  process  consistency  and 
consequent  effects  on  enterprise  acquisition  decisions. 

Time  of  interview: . _ Type  (circle  one):  Phone  or  in-person 

Date: _ Subject’s  MAJCOM: _ 

Subject’s  Position: _  Subject  Rank:  _ _ 

Subject’s  Product: _ Subject  Budget:  <$im  $iom  -  $ioom  -  $ib< 

Verbal  Statement  for  Participants 

Hello,  I  am  Captain  Eric  Perez,  a  student  at  the  Air  Force  Institute  of  Technology 
pursuing  a  Masters  in  Systems  Engineering.  Towards  this  effort,  I  am  conducting 
research  under  the  guidance  and  supervision  of  my  advisor,  Lieutenant  Colonel  Kyle 
Oyama.  Specifically,  you  are  being  asked  to  participate  in  a  short  interview  and  respond 
to  several  open-ended  questions.  Participation  in  the  interview  is  voluntary  and  there  is 
no  penalty  for  non-participation.  If  you  choose  to  participate  in  the  interview,  no 
personally  identifiable  information  will  be  collected  nor  will  your  unit/organization  be 
identified.  Our  correspondence  for  this  procedure  will  be  recorded  and  I  will  indicate 
when  the  recording  has  been  started  and  ended.  You  may  choose  not  to  answer  any  or  all 
of  the  questions.  The  recording  and  my  subsequent  analysis  will  be  kept  confidential  by 
storing  the  data  in  a  password-protected  file.  Participation  in  this  interview  and  data 
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collection  process  is  endorsement  of  your  consent.  Do  you  have  any  questions  regarding 
the  explained  terms  of  consent  and/or  what  you  are  be  asked  to  participate  in?  I  will  be 
investigating  the  consistency  of  project  risk  management  (PRM)  processes  across  the 
U.S.  Air  Force  and  its  effects  on  decisions  made  at  the  Air  Force  enterprise  level.  Thank 
you  in  advance  for  your  time. 

Background 

The  research  study  and  the  respective  interview  questions  will  be  investigating  the 
consistency  of  project  risk  management  processes  across  the  U.S.  Air  Force  and 
consequent  effects  on  decisions  made  at  the  Air  Force  enterprise  level. 

Definitions 

•  Project  Risk:  future  uncertainties  relating  to  achieving  program  technical 
performance  goals  within  defined  cost  and  schedule  constraints.  Defined  by  (1) 
the  probability  of  an  undesired  event  or  condition  and  (2)  the  consequences, 
impact,  or  severity  of  the  undesired  event,  were  it  to  occur  (Deputy  Assistant 
Secretary  of  Defense  for  Systems  Engineering  2014). 

•  PRM:  The  iterative  process  involved  with:  planning,  assessment  (identification 
and  analysis),  handling,  and  monitoring  of  risks  throughout  the  lifecycle  of  a 
project  or  program  (Under  Secretary  of  Defense,  2001). 

•  PRM  Consistency:  The  similarity  between  the  processes  used  by  different 
organizations  to  manage  the  effects  of  uncertainty  encountered  in  pursuit  of 
baselined  objectives. 

•  PRM  Processes:  The  means  by  which  uncertainty  is  handled  to  include: 
procedures,  methods,  and  techniques. 
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•  Unit:  For  the  purposes  of  this  interview,  a  unit  refers  to  a  group  of  individuals  to 
which  the  subject  interviewee  is  affiliated  with  that  share  common  objectives  in 
Air  Force  acquisitions. 

•  PRM  Steps:  The  different  phases  of  the  process  used  to  manage  risk  to  include: 
planning,  assessment  (identification  and  analysis),  handling,  and  monitoring. 

•  Implementation  methods:  techniques  used  to  carry  out  the  PRM  steps  unique  to 
the  interviewee’s  unit  to  include  tools,  types  of  communication,  assessment 
approach,  data  storage,  and  collaboration  methods. 

Questions 

1.  Primary  question:  What  is  your  involvement  with  the  PRM  processes  within  your 
unit? 

Follow-up  probe  question: 

a.  How  did  you  become  involved  in  this  way? 

2.  Primary  question:  How  does  the  emphasis  on  PRM  processes  at  your  current  unit 
compare  to  other  units  you’ve  been  involved  with  or  are  aware  of? 

Follow-up  probe  questions: 

a.  In  how  many  different  Units  have  you  been  involved  with  the  PRM  process? 

b.  Did  you  encounter  or  know  of  any  units  that  either  lacked  or  had  little  regard 
for  PRM?  If  so,  why  do  you  think  that  is? 

c.  Were  the  differences  in  emphasis  you  describe  driven  by  necessity  (i.e. 
differing  missions  or  schedule  pressures),  differing  guidance,  or  other 
reasons? 
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3.  Primary  question:  Between  the  different  unit  PRM  processes  you  are  aware  of,  what 
tends  to  be  inconsistent? 

Follow-up  probe  questions: 

a.  What  steps  make  up  your  unit’s  PRM  process  (Planning,  assessment 
(identification  and  analysis),  handling,  and/or  monitoring)?  Why  do  you  think 
these  specific  steps  are  used? 

i.  How  are  these  steps  different  than  other  units  and  what  do  you  think  is 
the  cause? 

ii.  Do  you  have  examples? 

iii.  Do  some  steps  get  more  emphasis  than  others?  What  do  you  think 
causes  the  difference? 

b.  What  tools  does  your  unit  use  to  implement  the  PRM  process?  Why  do  you 
think  these  specific  tools  are  used? 

i.  How  are  these  tools  different  than  other  units  and  what  do  you  think  is 
the  cause? 

ii.  Are  these  tools  available  to  the  right  personnel  (Ex:  Active  risk 
manager,  MSOffice)?  Why  do  you  think  these  specific  tools  are 
available? 

iii.  Are  the  tools  you  mentioned  adequate?  Why  or  why  not? 

iv.  On  which  portion  of  the  risk  management  process  are  the  tools  you 
mentioned  actually  used?  Why  or  why  not  certain  parts? 
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c.  What  techniques  does  your  unit  use  (Ex:  risk  matrix,  montecarlo)?  Why  do 


you  think  these  specific  techniques  are  used? 

i.  Do  these  techniques  differ  from  other  units  and  what  do  you  think  is 
the  cause? 

ii.  Do  you  think  these  techniques  are  adequate?  Why  or  why  not? 

d.  What  publications  does  your  unit  use  and  why  do  you  think  that  it  was 
selected? 

i.  Is  it  clear  which  publications  to  use? 

ii.  Are  the  publications  used  understandable  and  specific? 

iii.  Are  the  publications  used  adequate? 

e.  What  collaboration  methods  do  you  use  (Ex:  Delphi  technique,  story  board, 
meeting,  email)  and  why  were  they  chosen? 

i.  Are  the  methods  you  mention  different  than  in  other  units  you  know 
of?  Why  do  you  think  there  is  a  difference? 

ii.  Are  the  methods  you  mention  adequate  (i.e.  do  you  encounter  any 
challenges  such  as  bias,  strong  personalities,  or  group  think)? 

f.  What  methods  does  your  unit  use  to  report  risks  (Ex:  email,  meeting 
w/powerpoint)  and  why  do  you  think  it  is  done  that  way? 

i.  Who  are  the  stakeholders  that  receive  the  reports  (Ex:  customer, 
leadership,  peers)  and  why  do  you  think  they  are  slated  to  receive  this 
information? 

ii.  Does  the  status  of  who  is  receiving  the  reports  effect  the  risk 
assessments  that  are  reported? 
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g.  What  risk  identification  methods  (Ex:  email,  meetings,  delphi)  are  used  in 
your  unit  and  why  do  you  think  these  specific  techniques  are  used? 

i.  Are  there  differences  between  the  methods  used  in  your  unit  versus 
other  units  that  you  know  of,  and  if  so,  what  do  you  think  is  the  cause? 

ii.  Do  you  think  these  methods  adequately  identify  notable  risks? 

h.  What  are  the  analysis  techniques  used  to  assess  likelihood  and  impact  (Ex: 
best  estimate,  metrics  used,  strategic  to  fulfill  other  agendas)  and  why  do  you 
think  these  specific  techniques  are  used? 

i.  Are  there  differences  between  the  techniques  used  in  your  unit  versus 
other  units  that  you  know  of,  and  if  so,  what  do  you  think  is  the  cause? 

ii.  Do  you  think  these  techniques  produce  accurate  assessments? 

4.  Primary  question:  Based  on  the  inconsistencies  we  discussed,  how  do  you  think 
they  impact  risk  assessments,  if  at  all? 

Follow-up  probe  question: 

a.  Which  contributors  do  you  think  have  the  greatest  impact?  Are  the  impacts 
positive  or  negative? 

5.  Primary  question:  How  do  you  think  inconsistent  PRM  risk  assessments  impact 
enterprise  risk  management  (ERM)  decisions  at  the  macro  level  (Ex:  PEM  decisions 
on  which  projects  and  programs  to  fund)? 

Follow-up  probe  question: 

a.  How  do  you  think  inaccurate  PRM  processes,  and  ultimately  risk  assessments, 
relate  to  inconsistent  risk  assessments? 
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6.  Primary  question:  What  actions  do  you  think  could  be  taken  mitigate  the  effects  of 
inconsistent  PRM  risk  assessments  in  the  Air  Force? 

a.  Which  contributor(s)  to  inconsistent  Air  Force  PRM  should  receive  the  most 
focus? 
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Appendix  B 


Thesis  Interview  Follow-up  Likert  Questions 
Thesis:  Air  Force  Project  Risk  Management  Inconsistency  and  Potential  Solutions 
Study:  Investigation  of  Air  Force  Project  Risk  Management  process  consistency  and 
consequent  effects  on  enterprise  acquisition  decisions. 

1.  For  any  given  risk,  units  across  the  Air  Force  are  likely  to  arrive  at  the  same 
assessment  in  terms  of  the  probability  and  impact  that  would  be  reported. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

2.  Different  units  in  the  Air  Force  use  the  same  methods  and  techniques  to  identify 
project  risks. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

3.  Different  units  in  the  Air  Force  use  the  same  methods  and  techniques  to  assess 
project  risks. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

4.  There  is  pressure  to  provide  positive  news  about  risks  when  reporting  to 
supervisors  and  leadership. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

5.  Group  dynamics  such  as  group  think  and  strong  personalities  influence  your 
unit’s  risk  assessments. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

6.  There  is  clear  guidance  and  instruction  available  to  your  unit  on  how  to  carry  out 
risk  management. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 
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7.  Air  force  guidance  and  instruction  are  strictly  followed  in  your  unit. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

8.  The  techniques  and  methods  used  by  your  unit  to  carry  out  PRM  processes  are 
adequate  and  are  equally  as  effective  as  industry  best-practices. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 

9.  The  tools  used  by  your  unit  to  carry  out  PRM  processes  at  the  CGO/mid- 
management  level  are  adequate  and  are  equally  as  effective  as  industry  best- 
practices. 

Strongly  Disagree  -  Disagree  -  Uncertain  -  Agree  -  Strongly  Agree 
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